Jupyter: Trust notebook from command line
Jupyter uses a security model where you have to ‘trust’ a notebook before Jupyter will perform certain operations. While you can manually trust each notebook...
Posts by Category
til
Docker: Verify docker-compose configuration
Once you’ve edited your docker-compose.yml file, you can validate this file using the docker-compose config command that prints your resolved application con...
Python: The divmod() function
The [built-in] divmod() function returns a tuple containing the quotient and the remainder when argument1 (dividend) is divided by argument2 (divisor).
Git: Clone a specific branch
If you want a specific branch of a repo, rather than running:
Nix: Streaming gsutil transfers
Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be...
SQL: Calculate percentage of column
```sql Starting with a table. select * from sales; +——-+——+ | rep | sale | +——-+——+ | Bob | 15 | | Sally | 30 | | Peter | 15 | +——-+——+ Use a ...
SQL: Select a random sample
To pick a random subsample of 1,000 rows from a table:
Nix: Remove backspaces and tabs with col
Say you want to save the contents of the cp man page to a file.
Python: Get a notification via knockknock
There is a Python library called knockknock that allows you to get a notification when your training is complete or when it crashes during the process. Al...
Nix: Make a noise!
If you’re writing a script, and want it to make a noise to notify you when it’s done, look no further than SoX:
Pandas: Some notes on groupby
The count() aggregation function counts only non-null values. To count all values, whether null or non-null, use size.
Python: Use underscores as visual separators for numbers
Python 3.6 introduced (via PEP 515) the use of underscores as visual separators for numbers, in order to:
Website: How to create drafts in Jekyll
In case you haven’t noticed, I use Jekyll to create my blog from markdown files. And I typically write short posts, mostly TILs like this one. But I am tryin...
Nix: Echo and sudo for files owned by root
If you’re not root, and you want to modify a file owned by root (say /root/tarsnap-backup.sh), this won’t work:
Pandas: Three new functions
Towards Data Science on Medium has been a great source of tips so far, and this article is a great example that highlights the following functions new to me.
Pandas: Extra pd.options.display options
I’ve come across several pandas display options before, such as:
Nix: Sponge soaks up standard input and writes it to a file
I’m a huge fan of jq, a JSON parser for the command line. It’s great for reading elements from a JSON file:
Git: Dynamic identity
Sometimes I use my terminal for personal work, like this article. In those cases, I like my git commits to use my personal email address and not my work e...
Git: Sort branches by recency
When you type git branch, your branch list is sorted alphabetically by default. This isn’t super helpful. To sort your branches by their last commit date,...
Bash: Run entire shell script as root
Placing sudo in the shebang line of a shell script runs the entire thing as root. Useful for scripts designed to, e.g. automate system upgrades or package...
Pandas: Simplify filter expressions with between
You can reduce this filter:
Pandas: Regular expressions with str.contains
The pd.Series.str.contains method assumes that it is passed a regular expression for the pat input:
Python: Optimize logging of expensive debugging operations
Formatting of message arguments is deferred until it cannot be avoided. However, computing the arguments passed to the logging method can also be expensiv...
MacOS: How to validate your FileVault Recovery Key
I’m a big fan of keeping my laptop drive encrypted using macOS’ FileVault. As part of setting this up, you can set a recovery key in case you find yourself l...
Python: Fix issues using PyCharm and Docker
If you try to run a Docker container from within PyCharm, and you get something like:
Nix: Convert reStructuredText to Markdown
I’ve used pandoc a lot before to convert Markdown files to PDFs. I just found out it can also convert reStructuredText text files to Markdown format:
Bash: Get source directory of a script within the script
```bash #!/bin/bash
Mac: Logout a user from the command line
sudo launchctl bootout gui/$(id -u <username>) # or sudo launchctl bootout user/$(id -u <username>) Works on macOS 10.11.x or later.
Bash: Escaping strings easily
Enter a line of Bash starting with a # comment, then run !:q on the next line to see what that would be with proper Bash escaping applied.
Nix/Macos: See what a process is doing
To check what a long-rerunning process is doing on linux, use strace:
Nix: Count the number of fields in a record
To count the number of fields in a delimiter-separated text file, use awk:
Python: Get the most of floats
Similar to the int data type, floats also have several additional methods useful in various scenarios:
Python: Format text paragraphs with textwrap
Python’s textwrap module is useful for rearranging text, e.g. wrapping and filling lines.
Python: The SimpleNamespace Utility Class
The SimpleNamespace type from the types library provides an alternative to an empty class (class MyClass: pass) from which one can add and remove attribut...
Python: Random string generation with digits and upper-case letters
Here’s a nice one liner for generating a random string using just digits and upper-case letters:
Nix: Find the total size of files within a directory
Say you need to find the total size of jpegs within a directory with subdirectories. The following command has you covered:
Nix: Extract file extension from file name
If you need to know in a *nix script the extension of a file, try the following:
Python: Upgrading Homebrew packages using pip
Homebrew Python packages work by setting up their own package-specific virtual environments. This means you can upgrade them without waiting for the tap.
GitHub Actions: Skipping a step without failing
I wanted to have a GitHub Action step run that might fail, but if it failed the rest of the steps should still execute and the overall run should be treat...
ICS: Providing a ‘subscribe in Google Calendar’ link for a feed
If you provide your own custom generated ICS file hosted at a URL, it’s nice to be able to give Google Calendar users an easy way to subscribe to that fee...
Mac: Ask user for password via GUI
This function will use AppleScript to present a password entry dialog to make your scripts a little more user friendly:
Mac: Date and Time
List Available Timezones:
Mac: Restart or shutdown
You can restart or shutdown from the command line:
Nix: Stty - sane terminal settings
Restore sane shell settings, in case your shell session went insane because some script or application turned it into a garbled mess:
Mac: Gatekeeper via the command line
Add Gatekeeper Exception:
Nix: Fs_usage - View File System Usage
View a continuous stream of file system access info:
Nix: Mkfile - Create an empty file
Creates an empty 10 gigabyte test file:
Python: Salted Hash
The salt is just a randomly derived bit of data that you prefix or postfix your data with to dramatically increase the complexity of a dictionary atta...
Pandas: Combine Functions
pandas has two handy functions for combining DataFrames:
Pandas: Make Data Frame
pandas has a built-in function makeDataFrame() to return a DataFrame containing random floats. Note that this is using the private API, and the exact details...
Pandas: Read Clipboard
The pandas.read_clipboard() method is as simple as it sounds: it reads copy-pasted tabular data and parses it into a Data Frame. For instance, try running...
Pandas: Pipe function
Pandas introduced pipe() starting from version 0.16.2. pipe() enables user-defined methods in method chains.
Sklearn: Tree diagram
The plot_tree() function allows you to create a diagram of steps present in a decision tree model:
Sklearn: Pipeline diagram
Estimators can be displayed with a HTML representation when shown in a jupyter notebook. This can be useful to diagnose or visualize a Pipeline with many ...
Sklearn: Column transformations
The Scikit-learn pipeline has a function called ColumnTransformer which allows you to easily specify which columns to apply the most appropriate preproces...
Mac: Use Touch ID to Authenticate Sudo Commands
On newer MacBook Pros, you can add Touch ID as an acceptable method of authenticating sudo commands. This is done by editing the /etc/pam.d/sudo file:
Pandas: Named Aggregation
pandas>=0.25 supports named aggregation, allowing you to specify the output column names when you aggregate a groupby, instead of renaming. This will be e...
Mac: Remove quarantine flag from app
I encountered an issue with my favorite macos Markdown editor MacDown where macos Catalina was reporting the file as damaged. It turns out that Catalina has ...
Mac: Software Update from the Command Line
There are lots of Terminal commands that you can use to change or update your Mac’s OS. My favorite is this quick tip to download macOS updates and installat...
Nix: List files and group by extension
ls -X will group files by extension. — Unix tool tip (UnixToolTip) October 22, 2019
Science: Light v Sound
Roughly: Light travels about a foot per nanosecond, sound travels about a foot per millisecond. A factor of almost exactly a million. — Colin Wright ...
Bitbucket/GitHub: Create pull request from command line
Create a GitHub pull request from command line
Git: Blocked ssh port
I mostly clone GitHub and Bitbucket repositories using SSH URLs, so that I can protect this access with an SSH private/public keypair. Unfortunately, some fi...
Git: Stash tips
I’m a big fan of using git stash to shelve some changes in my repository so that I can move on to some other task. Here are some advanced git stash commands ...
MySql: Show full query in process list
SHOW PROCESSLIST
TSQL: Basic T-SQL
```sql – Show all databases. select name from master.sys.databases;
SQL: List Columns in Table
The ANSI standard way of listing all columns in a database table is:
Docker: Filtered system prune
Every now and then I run:
Spark: Date Arithmetic with Multiple Columns
Say you have a timestamp column created_at, and an integer column number that represents a number of weeks, how do you use the date_add function to calculate...
Spark: Count number of duplicate rows
To count the number of duplicate rows in a pyspark DataFrame, you want to groupBy() all the columns and count(), then select the sum of the counts for the ro...
Mac: Sync with time server (Mojave)
To check and update your system time in macos Mojave:
Docker: Set Timezone
To set which timezone your docker container should use, add the following to your Dockerfile:
Docker: Start Container as Root User
When sudo is not available in container, you can jump into a running container as root user using one of these commands:
Git: High-res plotswork on multiple branches simultaneously
Say you are on a feature branch, and want to make a bug fix in master. Rather than stashing your changes, or clone-ing the entire repository, you can create ...
Git: Untrack A File Without Deleting It
Generally when I invoke git rm <filename>, I do so with the intention of removing a file from the project entirely. git-rm does exactly that, removing ...
Git: Stashing Untracked Files
Normally when stashing changes, using git stash, git is only going to stash changes to tracked files. If there are any new files in your project that aren’t ...
Git: Snapshot
To save a snapshot of your current work in git, try this command:
Git: Stashing Only Unstaged Changes
If you have both staged and unstaged changes in your project, you can perform a stash on just the unstaged ones by using the -k flag. The staged changes will...
Git: Staging Stashes Interactively
The -p flag can be used with git stash, just as it is used with git add, for interactively staging a stash.
Git: Two ways of squashing commits
It is handy to squash down your commits before merging your PR with my-new-cool-feature. You can either squash them down by doing an interactive rebase like ...
Git: Specify the ssh key to use
The following methods will tell git which private key to use.
Git: Show The diffstat Summary Of A Commit
Use the --stat flag when running git show on a commit to see the diffstat summary of that commit. For instance, this is what I get for a recent commit to del...
Git: Show All Commits For A File Beyond Renaming
By including -- <filename> with a git log command, we can list all the commits for a file. The following is an example of such a command with some form...
Git: Interactively Unstage Changes
I often use git add --patch to interactively stage changes for a commit. Git takes me through changes to tracked files piece by piece to check if I want to s...
Git: Undo a Git Mistake
git reflog is a record of your actions in Git. With this command, you can undo almost any Git mistake.
Git: Resetting A Reset
Sometimes we run commands like git reset --hard HEAD~ when we shouldn’t have. We wish we could undo what we’ve done, but the commit we’ve reset is gone forev...
Git: Accessing A Lost Commit
If you have lost track of a recent commit (perhaps you did a reset), you can generally still get it back. Run git reflog and look through the output to see i...
Git: Reference A Commit Via Commit Message Pattern Matching
Generally when referencing a commit, you’ll use the SHA or a portion of the SHA. For example with git-show:
Git: Split Up a Commit, Rewrite History
When working on a branch with multiple commits, you can “go back in time” and revise previous commits any way you please.
Git: Git Log With Authors
In my never-ending quest to better summarize my work at the end of the day using computers, I discovered today the Git --author flag. It works like this:
Git: Git Log since
At the end of each day, I try to record what I did, to jog my memory during the next morning’s standup. This is a helpful aid:
Git: List Filenames Without The Diffs
The git show command will list all changes for a given reference including the diffs. With diffs included, this can get rather verbose at times. If you just ...
Git: List Different Commits Between Two Branches
There are times when I want to get a sense of the difference between two branches. I don’t want to look at the actual diff though, I just want to see what co...
Git: Last Commit A File Appeared In
In my project, I have a README.md file that I haven’t modified in a while. I’d like to take a look at the last commit that modified it. The git log command c...
Git: Ignore Changes To A Tracked File
Files that should never be tracked are listed in your .gitignore file. But what about if you want to ignore some local changes to a tracked file?
Git: List Most Git Commands
You can list most git commands by using the -a flag with git-help:
Git: Determine The Hash Id For A Blob
Git’s hash-object command can be used to determine what hash id will be used by git when creating a blob in its internal file system.
Git: Determine The Hash Id For A BlobGrep Over Commit Messages
The git log command supports a --grep flag that allows you to do a text search (using grep, obviously) over the commit messages for that repository. For the ...
Git: LFS Track
When you add a new type of large file to your repository, you’ll need to tell Git LFS to track it by specifying a pattern using the git lfs track command:
Git: LFS Pull
You can pull from a Git LFS repository using a normal git pull. No explicit commands are needed to retrieve Git LFS content. However, if the checkout fails f...
Git: LFS Prune
You can delete files from your local Git LFS cache with the git lfs prune command. This will delete any local Git LFS files that are considered ‘old’. An old...
Git: Migrate LFS hosting provider
To migrate a Git LFS repository from one hosting provider to another, you can use a combination of git lfs fetch and git lfs push with the --all option speci...
Git: Intro to Large File Storage (LFS)
Setting up Git LFS
Git: LFS Fetch
Git LFS typically only downloads the files needed for commits that you actually checkout locally. However, you can force Git LFS to download extra content fo...
Git: LFS Clone
Once Git LFS is installed, you can clone a Git LFS repository as normal using git clone. At the end of the cloning process Git will checkout the default bran...
Git: Clean Up Old Remote Tracking References
After working on a Git-versioned project for a while, you may find that there are a bunch of references to remote branches in your local repository. You know...
Git: List All Files Changed Between Two Branches
The git-diff command can help with finding all files that have changed between two branches. For instance, if you are at the HEAD of your current feature bra...
Git: Delete Remote Git Tags
Tagging releases with Git is a good idea. In case your tags get off track, here is how you delete a Git tag locally and on a remote:
Git: Configure to Use Single Key Presses in Interactive Mode
When staging changes in interactive mode (git add -p), you have a number of options associated with single keys. y is yes, n is no, a is this and all remaini...
Git: Diffing With Patience
The default diff algorithm used by Git is pretty good, but it can get mislead by larger, complex changesets. The result is a noisier, misaligned diff output.
Git: Amend Author Of Previous Commit
The author of the previous commit can be amended with the following command
Git: Delete All Untracked Files
Git provides a command explicitly intended for cleaning up (read: removing) untracked files from a local copy of a repository.
Git: Checkout Previous Branch
Git makes it easy to checkout the last branch you were on.
Git: Checkout Old Version Of A File
When you want to return to a past version of a file, you can reset to a past commit. When you don’t want to abandon a bunch of other changes, this isn’t goin...
Git: Grab A Single File From A Stash
In git, you can reference a commit SHA or branch to checkout differing versions of files.
Git: Use a file from another branch
Sometimes you just need one file from another branch. Sure you could git cherry-pick but then you’re dealing with commits. That sort of thing gets sticky fas...
Git: Clean Out All Local Branches
Sometimes a project can get to a point where there are so many local branches that deleting them one by one is too tedious. This one-liner can help:
Git: Intent To Add
Git commands like git diff and git add --patch are awesome, but their little caveat is that they only work on files that are currently tracked in the reposit...
Matplotlib: Save plot to file
It’s very straightforward to save a matplotlib figure to disk:
Docker: ‘Could not find’ with network prune
If you encounter the following error when starting a docker container:
Sklearn: Fix No Space Left on Device Error
While fitting a sklearn model, I encountered the following error:
Stats: Better descriptive statistics
Instead of the mean, use the median and/or the mode. Instead of the standard deviation, use the mean absolute deviation, the median absolute deviation, ...
Docker: Attach/Detach
To detach from a container, you hold Ctrl and press P, then Q. This only works if the container was started with both -t and -i.
AWS CLI: Modify volume size
Here’s how to modify the size of the volume attached to an EC2 instance “my_ec2”:
Travis: Skip unnecessary builds
Especially when you’re working with a large team with multiple Travis-enabled repositories, you’ll want to avoid running any unnecessary builds. The most ...
Matplotlib: Use LaTeX
Use LaTex in your plots:
Jupyter: Show contents of external script
%pycat shows you (in a popup) the syntax highlighted contents of an external file:
Jupyter: Share variables
To pass variables between notebooks, first store the variable using:
Jupyter: Set environment variables
You can set environment variables directly from the notebook without having to restart the kernel:
Jupyter: Output of all variables
If you try to see the output of more variables without explicitly writing print in front of each, only the last one gets outputted. With this, you get the...
Snowflake: Create table with comment
You can specify comments for a table, and even the columns in the table:
Pandas: Write DataFrame to table with to_sql
You can write DataFrames to a database table via a sqlalchemy connection as follows:
Pandas: sort_index
Dataframes have a new sort_index method to sort a dataframe by index. This is equivalent to the deprecated sort method with the columns argument set to `None.
Pandas: Use dtype to speed up reading with read_csv
By default, pandas will infer the data types of the columns when reading in a csv file. To speed up this read, you can specify the data types using the dtype...
Pandas: Use chunksize to iterate through files with read_Csv
Suppose you wish to iterate through a (potentially very large) file lazily rather than reading the entire file into memory.
Pandas: Extract days from a timedelta64
To extract the integer value of days from a numpy.timedelta64 in pandas, use dt.days to obtain the days attribute as integers.
Pandas: Inverse of boolean
In pandas, you can use the tilde (~) to flip bool values:
Pandas: Options
You can use the following functions to interact with the options in pandas:
Pandas: Option Context
If you want to temporarily change pandas options, instead of doing so manually as follows:
Pandas: Speed up merges
You can improve the speed of a merge by first specifying the key column of the merge as the index of your dataframes, and then using join instead of merge:
Pandas: Get Memory usage of DataFrame with info
The memory usage of a DataFrame (including the index) is shown when accessing the info method of a DataFrame. A configuration option, display.memory_usage, s...
Pandas: Groupby Not As Index
Let’s consider the following DataFrame:
Pandas: Iterating through groups
```python df = pd.DataFrame(np.random.randn(10,3),columns=list(‘ABC’))
Pandas: Get rows with max value of group
df.groupby('Sp').apply(lambda t: t[t.Count==t.Count.max()])
Pandas: DateOffset as a Frequency Increment
pandas’ DateOffset represents a regular frequency increment:
Pandas: Count number of non-NaN entries
The count() method returns the number of non-NaN values in each column. Similarly, count(axis=1) returns the number of non-NaN values in each row.
Pandas: Complex aggregation expressions
```python df = pd.DataFrame(np.random.randn(10,3),columns=list(‘ABC’))
Pandas: Calculate percentile ranking relative to another column
Say we have two columns of data representing the same quantity; one column is from training data, the other is from validation data. How can we efficiently g...
CompSci: GUIDs are not strings
GUIDs are not strings. They are numbers. We render them as strings for readability. We should not process them as strings. We should not pass them around ...
AWS CLI: List account aliases
If you want the URL for your sign-in page to contain your company name (or other friendly identifier) instead of your AWS account ID, you can create an al...
SQL: UNION v UNION ALL
UNION removes duplicate records, whereas UNION ALL does not.
Jupyter: High-res plots
You can enable high-resolution plots in jupyter notebook using the following configuration:
Numpy: Set number of decimal places using set_printoptions
> import numpy as np > np.set_printoptions(precision=4, ) > print(np.array([1.123456789])) [ 1.1235]
Mac: Emoji
Press the Command-Control-space to launch the characters palette. Then click on the Emoji icon in the sidebar on the left side of the Character window. You c...
Mac: Copy files intelligently with ditto
ditto is slightly more advanced but can be advantageous to ‘cp’ for several reasons, as it not only preserves ownership attributes and permissions but als...
Snowflake: Get_ddl statements
Get_ddl statements
Git: Using multiple worktrees
When working with multiple branches at the same time, people clone the whole git repository again.
Slack: Accessing Direct Messages from Disabled Users
You can’t actually delete a user in Slack. You can, however, disable a user. And the cool thing about disabled users is that you can still access the mess...
Bash: Append to variable without creating leading colon if unset
Say you need to append a directory to PATH, but only add a leading : if PATH is already set. The standard
Emacs: Kill line from anywhere on that line
Use M-shift-backspace.
LaTeX: Send a bit of LaTeX
You can use mathb.in to send someone a link to a bit of LaTeX.
Travis: Why is my build not running?
Sometimes you push to Travis CI and there is no new build. What to do in such case? Has Travis CI got your commits? Is the branch you were using disabled?...
Snowflake: Increment date and time values with dateadd
``` sql // Add two years to the current date: select dateadd(year, 2, current_date());
Matplotlib: Add title to collection of plots
Use matplotlib.pyplot.suptitle() to add a centered title to the figure:
Snowflake: Show file formats
To list the file formats for which you have access privileges, use:
Spark: Select all and add a new column
Here’s how to select all columns and add a new column.
Numpy: Extract days from timedelta64
To extract the integer value of days from a numpy.timedelta64, you divide it with a timedelta64 of one day:
Mac: Power shortcuts
Control-Command-Power/Eject will reboot the Mac instantly. Command-Option-Control-Power/Eject will shut it down. Command-Shift-Q will log off. Shift...
Python: Normalize text with unicodedata
Using the unicodedata Python module it’s easy to normalize any unicode data strings (remove accents etc):
Python: The last value printed
In the python REPL, _ is assigned the last value printed:
Python: Generate random but reproducible UUID with seed
I often generate UUIDs ( Universally Unique Identifiers ), but when I use these in testing, I want to do so reproducibly. Turns out you can do this using a s...
Python: Time the execution of a program
import time start_time = time.time() main() print("--- %s seconds ---" % (time.time() - start_time))
Python: SpooledTemporaryFile
The tempfile.SpooledTemporaryFile function operates exactly as TemporaryFile() does, except that data is spooled in memory until the file size exceeds the pa...
Python: Check available modules with pydoc
Find a list of all python modules installed on a machine by running the following command in a terminal:
Python: Private variables
To make Python treat a variable as pseudo-private, follow the convention of putting two underscores (i.e., __) at the beginning of the variable’s name, e.g.:
Python: Print without newline
To print a string without appending the usual newline, use the end parameter of the print function:
Python: Flushing while printing
Since Python 3.3, you can force the normal print() function to flush without the need to use sys.stdout.flush(); just set the flush keyword argument to Tr...
Python: Pickle in Python2 and Python3
The python3 pickle.load() function has optional keyword arguments that can be used to control compatibility support for pickle stream generated by Python 2:
Python: Sort lists naturally not lexicographically with natsort
When you try to sort a list of strings that contain numbers, the normal python sort algorithm sorts lexicographically, so you might not get the results that ...
Python: An improved tuple
A downside of plain tuples is that the data you store in them can only be pulled out by accessing it through integer indexes. You can’t give names to individ...
Python: MyPy variable annotations
In Python 3.6, variables (in global, class or local scope) can now have type annotations using either of the following two forms:
Python: Mutable default arguments
One of the most confusing moments for new developers is when they discover how Python treats default arguments in function definitions.
Python: Lambdas as lexical closures
A ‘lexical closure’ is a fancy name for a function that remembers the values from the enclosing lexical scope even when the program flow is no longer in that...
Python: Lambdas as function expressions
How do you define an function inline and then immediately call it? Like this:
Python: Parallel for loops
Joblib provides a simple helper class to write parallel for loops using multiprocessing. The core idea is to write the code to be executed as a generator ...
Python: Cache function output
Joblib traces parameters passed to a function, and if the function has been called with the same parameters it returns the return value cached on a disk.
Python: Invert a mapping
Given this dictionary:
Python: Hash a file
```python import hashlib
Python: Function disassembler
You can use Python’s built-in dis module to disassemble functions and inspect their CPython VM bytecode:
Python: Thousands Separator in Formatted Strings
It’s very easy to add thousands separators to numbers:
Python: Find the most common elements in an iterable
```python import collections c = collections.Counter(‘helloworld’)
Python: Specify requirements depending on python version using environment markers
You can use the environment markers to achieve this in requirements.txt since pip 6.0:
Python: Difference between append and ‘+=’
In many applications += and append are interchangeable, but for Python lists they are not.
Python: Deep copy a compound object
Assignment statements in Python do not copy objects, they create bindings between a target and an object. For collections that are mutable or contain muta...
Python: Nested Comprehensions
You probably know how to rewrite a for loop such as:
Python: Collect garbage
If you have a variable with a large memory footprint, you can force garbage collection using the gc Garbage Collector module:
Python: Class inheritance
You can check for class inheritance relationships with the issubclass() built-in:
Python: Check if string is null or empty
if mystring and mystring.strip(): print "not null and not empty string" else: print "null or empty string"
Spark: Calculating the length of a column with size
In Spark >= 1.5 you can use the size function to calculate the length of a column:
AWS CLI: Assuming a role
You can configure the AWS Command Line Interface to use a role by creating a profile for the role in the ~/.aws/config file. The following example shows a ro...
Matplotlib: Clearing a figure
```python import matplotlib.pyplot as plt
Matplotlib: Plot error as shaded region
```python import matplotlib.pyplot as plt import numpy as np
Matplotlib: Get current axis
matplotlib.pyplot.gca(**kwargs) gets the current Axes instance on the current figure matching the given keyword args, or create one.
Matplotlib: Use logarithmic scale
```python import matplotlib.pyplot as plt
Numpy: Return maximum of array ignoring NaNs using nanmax
>>> import numpy as np >>> np.nanmax([1, 2, np.nan]) 2.0
Nix: Check If A Port Is In Use
The lsof command is used to list open files. This includes listing network connections. This means I can check if a particular port is in use and what proces...
Nix: Check Ubuntu Version
Are you on Ubuntu? Want to know what version (release) of Ubuntu you are using?
Nix: Disk Speed Benchmark
In Linux, the dd command can be used for simple I/O performance measurements as follows:
Nix: CPU Benchmark
dd in conjunction with any stream-processing CPU-intensive program can be used as a simple CPU benchmark!
Nix: Saying Yes
Tired of being prompted for confirmation by command-line utilities? Wish you could blindly respond ‘yes’ to whatever it is they are bugging you about? The ye...
Nix: Max out CPU with Yes
If you want a quick and easy method to max out the usage of a CPU core, just use yes:
Nix: Where Are The Binaries
When I want to know where an executable is, I use which like so:
Nix: Get length of longest line with wc
wc -L shows the length of the longest line in a file.
Nix: Watch That Program
Have you ever been working in the terminal and found yourself repeating the same command many times? Delegate that work to the computer.
Nix: Monitor System Memory with vmstat
vmstat allows the user to monitor virtual memory statistics such as processes, memory, paging, block IO, traps , disks and cpu activity.
Nix: List contents of directories in a tree-like format
Why don’t I use this command more?
Nix: Duplicate pipe content
To duplicate the content while piping you can use the tee utility. One straightforward and useful example is that tee can be used to write to multiple fil...
Nix: Force cpus to run at max with stress
To make your 4 CPUs run flat out for 60 seconds:
Nix: Sorted human readable sizes
du -hsc * | sort -h
Nix: Sort In Numerical Order
By default, the sort command will sort things alphabetically. If you have numerical input though, you may want a numerical sort. This is what the -n flag is ...
Nix: See all versions of package in the archive
Use apt-cache policy <package>. For example:
Nix: Search Man Page Descriptions
You can use the apropos command with a keyword argument to search for that words occurrence throughout all the man pages on your system. For instance, invoki...
Nix: Search Files Specific To A Language
The ack command makes it easy to narrow the set of searched files to those of a specific programming language. For instance, if you have a rails project and ...
Nix: Save matching and non matching lines with awk
awk '/pattern/ {print $0 >"yes.csv"; next}{print $0 >"no.csv"}' input.csv
Nix: SSH With Port Forwarding
Use the -L flag with ssh to forward a connection to a remote server
Nix: Stay connected without an interactive SSH shell
You may need to connect to a remote location via SSH, but if the remote location doesn’t provide an interactive shell, the connection is most likely to dr...
Nix: Run local scripts remotely with SSH
To run a local script enki.sh on a remote machine, use:
Nix: SSH pipes
One of the benefits of piping is that you can use it over networks and it does wonders for data transfer. Note that half of the command is executed locally, ...
Nix: Killing A Frozen SSH Session
Whenever an SSH session freezes, I usually mash the keyboard in desperation and then kill the terminal session. This can be avoided though. SSH will listen f...
Nix: SSH Escape Sequences
To see the SSH Escape Sequences, hit <Enter>~?.
Nix: Reverse a String
Reverse a string with the rev command.
Nix: Repeat Yourself
Use the repeat command to repeat some other command.
Nix: Merge pdf files
You can use qpdf to merge pdf files into a single file as follow:
Nix: Monitor the progress of data through a pipe
You can use pv to monitor the progress of any pipe, by putting it between input/output pipes.
Nix: List parent pid with ps
The ps command, which stands for process status, is a great way to find different processes running on a machine. Information like their pid (process id) is ...
Nix: Create A File Descriptor with Process Substitution
Process substitution can be used to create a file descriptor from the evaluation of a shell command. The syntax for process substitution is <(LIST) where ...
Nix: PID Of The Current Shell
$ expands to the process ID of the shell. So, you can see the PID of the current shell with echo $$.
Nix: Standard output to clipboard buffer
Adding | pbcopy to the end of any command will send the standard output to your clipboard.
Nix: Open The Current Command In An Editor
If you are working with a complicated command in the terminal trying to get the arguments just right. Such as this curl:
Nix: Printing with lpr
Recently while trying to fix a printer I used lpr a bunch of times. It’s not exactly new to me, but never fails to surprise people when I use it.
Nix: List Of Sessions To A Machine
The last command is a handy way to find out who has been connecting to a machine and when.
Nix: Kill Everything Running On A Certain Port
You can quickly kill everything running on a certain port with the following command.
Nix: Install Packages From A Specific Repository
Here’s how to install a package from a specific repository:
Nix: Find MAC address of network interfaces with ifconfig
To get the MAC addresses of all your interfaces, grep the HWaddr from the ifconfig command:
Nix: Search History
Often times there is a very specific command you have entered into your bash prompt that you need to run again. You don’t want to have to type it again and s...
Nix: Pattern-matched search of your history
In bash, !?foo will repeat the most recent command that contained the string ‘foo’.
Nix: Last Argument Of The Last Command
You can use !$ as a way to reference the last argument in the last command. This makes for an easy shortcut when you want to switch out commands for the same...
Nix: Global Substitution On The Previous Command
Let’s say we just executed the following command:
Nix: Hexdump A Compiled File
The hexdump unix utility allows you to dump the contents of a compiled/executable file in a _readable_ hexadecimal format. Adding the -C flag includes a side...
Nix: Only Show The Matches
Tools like grep, ack, and ag make it easy to search for lines in a file that contain certain text and patterns. They all come with the -o flag which tells th...
Nix: List Names Of Files With Matches
I often use grep and ag to search for patterns in a group or directory of files. Generally I am interested in looking at the matching lines themselves. Howev...
Nix: Grep For Multiple Patterns
You can use the -e flag with the grep command to search for a pattern. Additionally, you can use multiple -e flags to search for multiple patterns. For insta...
Nix: Grep For Files Without A Match
The grep command is generally used to find files whose contents match a pattern. With the -L (--files-without-match) flag, grep can be used to find files tha...
Nix: Command Line Length Limitations
The other day I tried to run a rm command on the contents of a directory with a LOT of files.
Nix: Find process preventing file deletion with fuser
Sometimes when deleting a file, the error “File is already in use” is encountered, with further trouble locating the process using the file.
Nix: Find Newer Files
Use the -newer flag with the name of a file to find files that have a newer modification date than the named file.
Nix: Exclude A Directory With Find
Using find is a handy way to track down files that meet certain criteria. However, if there are directories full of irrelevant files, you may end up with a l...
Nix: Upgrading Ubuntu
I recently discovered that my Linode box was running a fairly old version of Ubuntu. Because it is a remote box that I SSH into, there is no graphical user i...
Nix: Do Not Overwrite Existing Files
When using the cp command to copy files, you can use the -n flag to make sure that you do not overwrite existing files.
Nix: Determine The IP Address Of A Domain
The dig (domain information grouper) command can be used to get more information about a domain name. To discover the IP address for a given domain, invoke d...
Nix: Curling For Headers
If you want to inspect the headers of a response from some endpoint, look no further than a quick curl command. By including the -I flag, curl will return ju...
Nix: Curling With Basic Auth Credentials
I often use curl to take a quick look at the responses of particular endpoints. If I try to curl a URL that is secured with HTTP Basic Authentication, this i...
Nix: Convert tabs to/from spaces
The command expand in GNU coreutils converts tabs in each input file to spaces. The command unexpand does the reverse, converting spaces in each input files ...
Nix: Convert file contents to lower case with tr
To convert the contents of a file to all lower case, you can use:
Nix: Combine multiple consecutive blank lines into one
cat --squeeze-blank (or cat -s for short) will suppress repeated empty output lines.
Nix: Change Default Shell For A User
You can change the default shell program for a particular unix user with the chsh command. Just tell it what shell program you want to use (e.g. bash or zsh)...
Nix: Cat A File With Line Numbers
You can quickly view a file using cat
Spark: Orderby Partitioning
Remember that orderBy uses the number of partitions specified by spark.conf.get("spark.sql.shuffle.partitions"). The default for this is 200. Can change manu...
Spark: Materializing and Unpersisting Cache
If you want to reuse a dataframe df without having to recreate it, you can use df.cache() to tell Spark to keep the dataframe in memory.
Tmux: tmux in your tmux
If you are running tmux locally and you shell into another machine to access tmux remotely, you will suddenly find yourself in tmux inception. You will have ...
Tmux: Adjusting Window Pane Size
In tmux, the size of window panes can be adjusted incrementally with the resize-pane command. For instance, to resize a pane in any direction (left, down, up...
Tmux: Rename The Current Session
If you’ve created an unnamed tmux session or you no longer like the original name, you can open a prompt to change it by hitting
Tmux: Pane Killer
The current pane can be killed (closed) using the following key binding:
Tmux: Paging Up And Down
When in copy mode (<prefix>[), you can move the cursor around like you would in vim with the directional keys (hjkl). This works fine until you want to...
Tmux: Open New Window With A Specific Directory
When you initially start a tmux session, the default directory is based off of whatever the current working directory was. Any subsequent windows opened with...
Tmux: Create A New Session In A New Server
Any tmux command will, by default, be invoked against the default server. You can instruct tmux to perform commands against a different server with the -L fl...
Tmux: Create A Named tmux Session
When creating a new tmux session
Tmux: List Sessions
Not sure if tmux is running or, if it is, which sessions are available? You can list all the currently running sessions right from the command-line.
Tmux: List All Key Bindings
There are a couple ways to list all the tmux key bindings. If you are not currently in a tmux session, you can still access the list from the terminal with
Tmux: Kill The Current Session
When you are done with the current tmux session and you no longer need it, you can simply kill it. You can do so within the session with the following comman...
SQL: Word Count for a Column
Assuming I have a database with a posts table:
SQL: Cleanup Databases
drop database foo_development;
SQL: Day Of Week For A Date
Given a date in PostgreSQL
SQL: Day Of Week By Name For A Date
By using the to_char() function with a date or timestamp, we can determine the day of the week by name (e.g. Monday). For instance, to determine what day tod...
SQL: Count Records By Type
If you have a table with some sort of type column on it, you can come up with a count of the records in that table by type. You just need to take advantage o...
PSQL: Checking The Type Of A Value
The pg_typeof() function allows you to determine the data type of anything in Postgres.
PSQL: Terminating A Connection
Consider the scenario where you are trying to drop a database, but there are existing connections.
PSQL: List Connections To A Database
The pg_stat_activity table can be used to determine what connections there currently are to the PostgreSQL server and to a particular database. To see the pr...
PSQL: Sleeping
Generally you want your SQL statements to run against your database as quickly as possible. For those times when you are doing some sort of debugging or just...
PSQL: Get The Size Of An Index
Want to get an idea of how much disk space that additional index is taking up?
PSQL: Restore a database
To restore the dump, create a fresh database and then use pg_restore:
PSQL: Get The Size Of A Table
With the pg_relation_size() function, we can get the size of a given table. For instance, if we’d like to see the size of the reservations table, we can by e...
PSQL: Dump a database
Using the pg_dump with the -Fc flag will create a dump of the given database in a custom format. The output of this command can be redirected into a file (th...
PSQL: Get The Size Of A Database
If you have connect access to a PostgreSQL database, you can use the pg_database_size() function to get the size of a database in bytes.
PSQL: Change The Current Directory
When you start a psql session, your current directory is what psql will use as its current directory. This is important for meta-commands that use relative p...
PSQL: Auto Expanded Display
By default, postgres has expanded display turned off. This means that results of a query are displayed horizontally. At times, the results of a query can be ...
PSQL: Sets With The Values Command
You can concisely create sets of values in PostgreSQL using the values command.
PSQL: Who Is The Current User
You can determine the current user of a psql session by selecting on the current_user
PSQL: Use Argument Indexes
In Postgres, each of the arguments you specify in a select statement has a 1-based index tied to it. You can use these indexes in the order by and group by p...
PSQL: Use unlogged tables for caches
Using a Postgres table for caching? You might want to try making it unlogged.
PSQL: Types By Category
Postgres has many types, each of which fall into a particular category. These categories include Array, Boolean, String, Numeric, Composite, etc. Each of the...
PSQL: Truncate Tables With Dependents
If you have tables A and B where B has a foreign key referencing A, then trying to truncate A will result in something like this:
PSQL: Truncate All Rows
Given a postgres database, if you want to delete all rows in a table, you can use the DELETE query without any conditions.
PSQL: Turn Timing On
When digging around your database and running queries, it is helpful to have an eye on the speed of those queries. This can give insight into where there are...
PSQL: List All Rows In A Table
Perhaps the more common way to list all rows in a table is with the following select command:
PSQL: Limit Execution Time Of Statements
You can limit the amount of time that postgres will execute a statement by setting a hard timeout. By default the timeout is 0 (see show statement_timeout;) ...
PSQL: Limit Execution Time Of StatementsSpecial Math Operators
Postgres has all the mathematical operators you might expect in any programming language (e.g. +,-,*,/,%). It also has a few extras that you might not be exp...
PSQL: Find The Location Of Postgres Config Files
If you can connect to your database with psql, then you can easily find the location of your Postgres config files. After connecting, I can ask Postgres to s...
PSQL: Find The Data Directory
Where does postgres store all of the data for a database cluster? Well, in its data directory. Where exactly that data directory is can depend on how the dat...
PSQL: Configure The Timezone
Running show timezone; will reveal the timezone for your postgres connection. If you want to change the timezone for the duration of the connection, you can ...
PSQL: Set A Seed For The Random Number Generator
In PostgreSQL, the internal seed for the random number generator is a run-time configuration parameter. This seed parameter can be set to a particular seed i...
PSQL: Send A Command To psql
You can send a command to psql to be executed by using the -c flag
PSQL: Use a psqlrc File For Common Settings
There are a handful of settings that I inevitably turn on or configure each time I open up a psql session. I can save myself a little time and sanity by conf...
PSQL: A Better Null Display Character
By default, psql will display null values with whitespace. This makes it difficult to quickly identify null values when they appear amongst a bunch of other ...
PSQL: String Contains Another String
You can check if a string contains another string using the position function.
PSQL: Salt And Hash A Password With pgcrypto
The pgcrypto extension that ships with PostgreSQL can be used to do a number of interesting things. This includes functions for doing salted password hashing...
PSQL: Generating UUIDs With pgcrypto
If you check out the docs for the uuid-ossp extension, you’ll come across the following message.
PSQL: Compute Hashes With pgcrypto
The pgcrypto extension that comes with PostgreSQL adds access to some general hashing functions. Included are md5, sha1, sha224, sha256, sha384 and sha512. A...
PSQL: List Various Kinds Of Objects
Our PostgreSQL database can end up with all kinds of objects: tables, sequences, views, etc. We can use a variety of psql meta-commands to list the different...
PSQL: List Database Users
Within psql, type \du to list all the users for a database and their respective permissions.
PSQL: List Database Objects With Disk Usage
I’ll often times use \d or \dt to check out the tables in my database. This shows the schema, object name, object type (e.g. table), and owner for each.
PSQL: List All The Databases
There are two ways to list all the available databases. The first is a psql only command:
PSQL: Insert Just The Defaults
If you are constructing an INSERT statement for a table whose required columns all have default values, you may just want to use the defaults. In this situat...
PSQL: Generate Series Of Numbers
Postgres has a generate_series function that can be used to, well, generate a series of something. The simplest way to use it is by giving it start and stop ...
PSQL: Export Query Results To A CSV
Digging through the results of queries in Postgres’s psql is great if you are a programmer, but eventually someone without the skills or access may need to c...
PSQL: Clear The Screen In psql
The psql interactive terminal does not have a built-in way of clearing the screen. What I usually do if I really need the screen cleared is quit, run clear f...
PSQL: Storing Emails With citext
Email addresses should be treated as case-insensitive because they are. If a user is trying to sign in with their email address, we shouldn’t care if they ty...
PSQL: Getting A Slice Of An Array
Postgres has a very natural syntax for grabbing a slice of an array. You simply add brackets after the array declaring the lower and upper bounds of the slic...
PSQL: Defining Arrays
In postgres, an array can be defined using the array syntax like so:
PSQL: Renaming A Table
Using the alter table command in PostgreSQL, you can rename an existing table. This command will also update any references to the table such as via foreign ...
PSQL: Restart A Sequence
In postgres, if you are truncating a table or doing some other sort of destructive action on a table in a development or testing environment, you may notice ...
PSQL: Determining The Age Of Things
In PostgreSQL, we can determine the age of something (or someone) by passing a timestamp to the age function.
Mac: Resizing Both Corners Of A Window
Hold the option key while resizing a corner of a window and it will simultaneously and equivalently resize the opposite corner.
GitHub: Link to headers in READMEs
Anytime you add a header to a markdown file, GitHub attaches an href with its downcased name. ‘JavaScript’ receives a link to #javascript, for instance.
GitHub: Exclude Whitespace Changes From GitHub Diffs
If you run a tidy ship and use plugins like vim-spacejam, then whitespace changes cluttering up your git diffs probably isn’t much of an issue.
GitHub: Add Emoji To GitHub Repository Description
Add Emoji To GitHub Repository Description
Bash: Partial String Matching In Bash Scripts
To compare two strings in a bash script, you will have a snippet of code similar to the following:
Bash: Directional Commands
You can move the cursor without arrow keys. Here is the keyboard equivalent for each.
Bash: Jump To The Ends Of Your Shell History
There are all sorts of ways to do things in your shell environment without reaching for the arrow keys. For instance, if you want to move up to the previous ...
Homebrew: Switch Versions of a Brew Formula
If you’ve installed a couple versions of a program via brew and you’d like to switch from the currently linked version to the other installed version, you ca...
Nix: The cron schedule expression editor
Do you always forget how to read/write cronjobs? Check out crontab.guru - the cron schedule expression editor.
posts
The evolution of US gun violence
Heather Cox Richardson is an American historian and professor of history at Boston College. She has a wonderful newsletter Letters from an American worth sub...
The social contract of open source
I’m a huge fan of open source software. Much of my work and play involve using it, either directly or indirectly. And so I’m a big proponent of people giving...
Entropy Explained, With Sheep
All Physics students learn the Second Law of Thermodynamics, that entropy always increases. But not all such students understand why this is. To do this, let...
The true meaning of work-life balance
Some advice from a parent dealing with the death of his young son:
Debunk Flat Earthers
Carl Sagan debunks Flat Earthers using nothing more than a piece of cardboard:
How to talk to children
As a parent of two small children, I have to constantly remind myself that some positive or negative event that seems trivial to me may be incredible or deva...
How to create a healthy society
Microsoft Research’s danah boyd has been given an award by the Electronic Frontier Foundation, and gave a magnificant speech on her experience as a women in ...
How to Use Astrophysics to Solve Earthbound Problems
The cross-pollination that naturally occurs when people move along a diverse career path is of great benefit to the areas they find themselves in. This was e...
How to be Black
I just finished Baratunde Thurston’s How to be Black, a wonderful “satirical guide to race issues – written for black people and those who love them”.
The Importance of a Cup of Tea
A discriminating palate leads to novel rigorous statistical methods
Are categorical variables getting lost in your random forests?
I’ve been one-hot encoding categorical variables for as long as I have been using sci-kit learn. It turns out that you can lose a lot of predictive power thi...
Gathering weak npm credentials
We all know the importance of strong passwords, don’t we?
Canonical Correlation Analysis for Analyzing Sequences of Medical Billing Codes
Due to the vast number of medical billing codes, it is generally infeasible to generate machine learning features from them as one-hot vectors. The paper Can...
Explaining complex machine learning models with LIME
Another nice write-up on the use of Local Interpretable Model-Agnostic Explanations (LIME) to explain complex machine learning models.
SHAttered
Awesome work to demonstrate how to deliberately cause a SHA-1 collision.
The legends of mathematics that almost never were
It’s always made me sad when people tell me they dislike mathematics. I always wonder if it was the subject or their teachers that they disliked…
The Truth About Bad Science
This wired.com article on ‘bad science’ speaks to me on so many levels. It hurts my soul to see how many published studies are not reproducible.
Why Should I Trust You?: Explaining the Predictions of Any Classifier
If you can’t understand how a model makes a prediction, how can you trust that prediction?
Death by Diagnosis on Freakonomics
Q: What’s the number-one problem in healthcare? A: I think the number-one problem is we don’t measure performance. We don’t measure the outcomes of patien...
Making machine learning models interpretable
This month, the 20th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning held a special session on “Interpretab...
The relativity of raw data
Data scientists often say that they want access to the ‘raw data’ – but what does that term mean?
The Friendliest of Fire
How on earth could an American pilot get a Distinguished Flying Cross for shooting down an American plane? The Friendliest of Fire (via Now I Know) tells all.
Hello World!
In the proud tradition of programmers everywhere, I use my first blog post to say “Hello World!”.
tips
Jupyter: View notebook in terminal
You can use the command line tool jut to view a Jupyter notebook in the terminal.
Nix: Play with jq with jq-play
Use jq-play to dry-run your jq filter and optimize any way you want.
Match arguments to help text
explainshell.com is a great website tool where you enter a shell command line to see the help text that matches each argument.
HTTP: Simulate unexpected network conditions with deelay.me
Slow loading resources (images, scripts, etc) can break your application. With this proxy you can simulate unexpected network conditions when loading a sp...
Machine Learning for Product Managers
In a previous post, I discussed the importance of learning how to properly communicate Data Science to maximize the impact of your work. Product Managers are...
Communicating Data Science with impact
One of the major differentiators between a new Data Scientist and a more experienced one is how the more senior practitioner spends a lot of time understandi...
Data problems
Another nice Medium post from Benjamin Obi Tayo has a good summary of the types of issues you should always be mindful of when you get a new data set:
Model error quantification
When we train a machine-learning model, we almost always report some performance metric, such as accuracy, recall, or F1-score.
Probabilistic interpretation of AUC
The area under the ROC (receiver operating characteristic) curve, or AUC, is a popular and robust metric for machine learning classification. However, one is...
The Modal American
When trying to characterize a dataset, we often reach for the old standby: the mean of each property. If we give it some more thought, we might consider usin...
Combating Fake News With a Smartphone
Automatically [add] extra digital proof data to all photos and videos you take.
Unlearning descriptive statistics
Top tips on better descriptive statistics:
The Data Visualisation Catalogue
Looking for inspiration for your data viz project? Can’t remember what a particular visualization is called? Check out the Data Visualisation Catalogue.
tres
Minnow Telescope Finds Massive Planet
Since ancient times, mankind has studied the sky and wondered what the ‘wandering stars’ (planets) might be. In the last two decades, we have found hundreds ...
The Little Telescope that Could
In the shadow of the ‘Big Eye’, this is the Little Telescope That Could…
Pandas: Split string column into separate columns
I use pd.Series.str.split() a lot for feature engineering, but I recently learned of a useful option expand for this function. This option if set to the non-...