multi-file sort NCSA combined based log files

problem: you want to grep and sort multiple NCSA combined based log files

For example, you have a dir with 30 log files for an apache2 vhost, some uncompressed, some compressed (logrotated) etc. Months and years might be arbitrary. Lets say you want to concatenate all the log entries, grep/awk a specific keyword such as year, and then sort the entries based on the NCSA combined timestamp format, which doesn't naturally sort as single key/field... 

The NCSA timestamp format:  date +'[%d/%b/%Y:%H:%M:%S %z]' yields: [18/May/2026:22:12:35 +0100] 

So 👆 this timestamp format is actually spanning two fields and requires specific subfield sorting.

impact: manual repetition required for each file

Without a multi-file pipeline, the operator will:

1. have to check each file individually
2. not have a single overview e.g. less buffer or concatenated output file
3. not have a single grepable text stream for further filtering/discovery

solution: zgrep pipeline

1. find ... -exec ... {} + efficiently avoids any glob argument limits
2. zgrep -Fhi automatically handles both (un)compressed files, omitting filenames, ignoring case and  searching for a fixed string e.g. file.php
3. awk filters based on one or more column values, e.g. timestamp year
4. LANG=C sort -s performs stable sorting using the C locale.
   Multiple -k options define a cascading sort hierarchy, where each option's argument specifies the sub-field position and sort type (numeric/month/etc).
5. The less -S pager provides a scrollable and searchable buffer for overview and further discovery

# subshell to avoid changing shells pwd/cwd and provides a single text stream to awk
( cd /var/log/apache2/sub.domain.tld && \
  find . -maxdepth 1 -name 'access.log*' -exec zgrep -Fhi file.php {} + ) \
| awk '$4 ~ /\/2026:/' \
| LANG=C sort -s -k4.10,4.13n -k4.6,4.8M -k4.3,4.4n -k4.15,4.22 -k5.3,5.7n \
| less -S

This scenario reminded me of my 2010 post on Parsing NCSA combined log format - working with columns.

Tested with sort (GNU coreutils) 9.1.

Windows: no easy way to view a list of arbitrary file paths

problem: You have a list of arbitrary file paths and no easy way to view them

Let's say you're doing a file search on a cifs/smb share to find files with certain attributes or naming patterns.

You'd like to be able to view or preview the the file paths efficiently one by one, but the file paths are arbitrary. This means:

1. The files are not all in the same directory.
2. Some files may be in the same directory AND we want to ignore other files in such a case.
   

impact: this makes viewing the files difficult

Windows does not have native functionality to support viewing such a list of files.

solution: make a loop with cygstart

Cygwin can help here with its cygstart command.

# define pause function
function pause() { read -n1 -p"$@" </dev/tty; }
# export pause function
export -f pause

while IFS= read -r line; do printf "%q\n" "$line"; done < ../path/to/list_of_file_paths.txt | tr \\n \\0| xargs -I{} -0 -n1 -- sh -c 'echo "$1"; pause "press any key to cygstart the file, or CTRL+C to abort"; cygstart "$1"' cygstart_loop {}

# 💡 the .txt file is expected to contain file paths one per line, if some file paths contain a line break this logic needs to be updated to handle such a scenario.

# 👆 breakdown on the above loop
# 1. read the input .txt file line by line - expects file paths one per line

# 2. prints lines with %q format, which makes the output safe/escaped for shell input

# 3. translates newlines to zero/null bytes - assumes that no filename contains a newline
     note: this may not be strictly necessary because we have used %q format BUT it does
     explicitly document that we are working with records separated by new lines,
     converting the new lines to zero/null bytes, and xargs is running in -0 mode.
     xargs -0 removes any ambiguity in the record separator and mitigates file path special characters causing interpretation issues.
     
# 4. use xargs to run a simple sh script to prompt the user if they would like to cygstart the given file.
     the user is shown the interpreted file path with the chance to abort.
     if the user does not abort - cygstart will attempt to open the file path using the file types default program for the file type.
     note: this method should be mitigate command injection exploits, see: https://unix.stackexchange.com/a/156010/19406
     note: cygstart_loop is the name of the ad-hoc sh script

citation:

Props to:
Stéphane Chazelas @ Stack Exchange / Unix & Linux / their detailed answer on using shell input safely.

Secure defaults for sshd_config including Multi-Factor-Authentication (MFA)

I posted a gist documenting secure defaults for sshd_config which includes Multi-Factor-Authentication (MFA). The configuration strategy aims to mitigate various attacks and exploits, disables password auth, and forces users to use MFA. 

You can find the gist here: https://gist.github.com/kyle0r/eb6b9e16ad6366ffa9692169906f128a

Minecraft Windows 10 edition (app) - edit player name for offline multiplayer

problem: you wish to change your local multiplayer player name

In the Windows 10 edition (not Java), there doesn't seem to be an obvious, straightforward way to change the player name for offline LAN multiplayer.

impact: you cannot change your player name

solution: edit options.txt file

When you run these steps, a txt file will open and you can edit the  mp_playername option to your desired player name for local LAN multiplayer games.

Tested on Minecraft version 1.19.81.

The steps are as follows:

1. Close Minecraft if open.
   
2. Select and copy CTRL+C this path to the options.txt file:

   "%LocalAppData%\Packages\Microsoft.MinecraftUWP_8wekyb3d8bbwe\LocalState\games\com.mojang\minecraftpe\options.txt" 

3. WIN+R (open the run prompt)
   
4. CTRL+V (paste the path)
5. ENTER (your default editor for .txt files should open e.g. Notepad)
   
6. Edit  mp_playername option (white-space and certain characters likely restricted)
7. CTRL+S (save the file)
8. Load Minecraft to see the change.
   

citation:

I didn't find a direct source for this, but some things I read while researching solutions gave me the idea to search within configuration files that might contain a solution. I found options.txt and the change worked.

Archiving Smarter Every Day episodes

I wrote up my steps for grabbing online media content (audio and video) from content platforms such as YouTube using the yt-dlp utility. I took the Smarter Every Day channel as an example of important intellectual content and recorded some related tutorials.

You can find it hosted on Coda here: handy-to-know-shizzle/archiving-smarter-every-day-episodes.

Bracketed paste - prevents pasting commands into vim

Problem: Why can't I paste commands into vim?

I've had this problem in at least two environments I work in. It came up against just recently, so I'm taking a moment to document it. Let us say you have the following in your clipboard:

:set tabstop=4 shiftwidth=4 expandtab

So you are inside vim and you press your usual paste keystroke for your terminal e.g. SHIFT + INSERT. You expect the command to appear in the vim command line area, but instead something else happens and maybe some of the clipboard content is pasted into the buffer instead?

😡🤬

Impact: frustration and lost productivity...

Everyone hates to lose their flow state because of annoying issues like this. At least from my 20+ years of experience with Linux, it's a non-standard behaviour (or perhaps a change in the old/legacy behaviour).

Solution: 

It is possible that this issue only affects xterm-like terminals. I use mintty heavily in my daily workflows.

This post on the Stack Exchange vim site captures the problem / solution. Its  straightforward - at runtime and/or in your ~/.vimrc use the following:

" disable bracketed-paste - which prevents pasting commands into vim
set t_BE=

It helps to understand bracketed-paste: https://en.wikipedia.org/wiki/Bracketed-paste. In addition it helps to understand the relevant bracketed-paste sections of the vim manual on bracketed-paste.

Citation:

Props to: the people on the Stack Exchange post.

Windows smb share file permissions cache / race condition issue

Problem: windows client cannot access a file on an smb share but the permissions are correct on the server

Client: Windows 10 22H2 (OS Build 19045.2006) smb dialect 3.1.1
Server: Linux - Debian 10 - buster - smbd version 4.9.5

I had an issue that a specific file on an smb share connected via windows client was inexplicably out of sync with the server permissions and ACL's. The file could be listed but read/write permission was denied. Using cygwin to list the file permissions showed disparity between the client and the server.

The file had been written by a Linux client and the Windows client had inexplicable permissions issues. Explorer, and other programs demonstrated the permissions issues. Here is how it looked like from a cygwin prompt on the windows client:

user@node-5900x //omv.blah.local/share
$ file merge.mp4 ; touch merge.mp4
merge.mp4: regular file, no read permission
touch: cannot touch 'merge.mp4': Permission denied

Cross-checking the permissions and ACL's on the server and another Linux client - everything seemed fine. Explicitly touching, chown and chmod'ing the file didn't help to wake up the windows client to see the correct permissions. Restarting smbd on the server also didn't seem to help.

Creating more new files on the Linux client and checking them on the Windows client - everything was OK... It was this specific file that was having issues.

Not sure if its related but the command that created the file (the writing binary) on the Linux node was as follows:

ffmpeg -i merge.mkv -strict experimental -c copy merge.mp4

Impact: client unable to work with the file

You could say this was a kind of service outage for the client. This would obviously impact the productivity of the person(s) working on the client.

Solution: restart the workstation service on client

The smb connection was not listed with net use, so net use was not the right approach to delete the session/connection for the share in this case. I found a few posts suggesting that a restart of the clients workstation service would clear out sessions/credentials and could solve issues - it did. I have a feeling I've used this approach in the past - it was just too long ago to remember it.

Prior to restarting the service the smb connections list looked like this via elevated PowerShell:

PS C:\WINDOWS\system32> Get-SmbConnection

ServerName       ShareName UserName         Credential       Dialect NumOpens
----------       --------- --------         ----------       ------- --------
omv.blah.local   share     NODE-5900X\user  NODE-5900X\user  3.1.1   10

To restart the workstation service - from an elevated cmd prompt:

net stop workstation && net start workstation

💡 Its important to ensure all explorer and other programs using the share are closed, otherwise this solution might not work as advertised.

Alternative: logging off the windows user and/or restarting the windows node would likely of also resolved this issue. However those approaches are disruptive and sometimes highly undesirable because they can impact peoples workflow and productivity.