Monday, January 30, 2023

Windows smb share file permissions cache / race condition issue

Problem: windows client cannot access a file on an smb share but the permissions are correct on the server

Client: Windows 10 22H2 (OS Build 19045.2006) smb dialect 3.1.1
Server: Linux - Debian 10 - buster - smbd version 4.9.5

I had an issue that a specific file on an smb share connected via windows client was inexplicably out of sync with the server permissions and ACL's. The file could be listed but read/write permission was denied. Using cygwin to list the file permissions showed disparity between the client and the server.

The file had been written by a Linux client and the Windows client had inexplicable permissions issues. Explorer, and other programs demonstrated the permissions issues. Here is how it looked like from a cygwin prompt on the windows client:

user@node-5900x //omv.blah.local/share
$ file merge.mp4 ; touch merge.mp4
merge.mp4: regular file, no read permission
touch: cannot touch 'merge.mp4': Permission denied

Cross-checking the permissions and ACL's on the server and another Linux client - everything seemed fine. Explicitly touching, chown and chmod'ing the file didn't help to wake up the windows client to see the correct permissions. Restarting smbd on the server also didn't seem to help.

Creating more new files on the Linux client and checking them on the Windows client - everything was OK... It was this specific file that was having issues.

Not sure if its related but the command that created the file (the writing binary) on the Linux node was as follows:

ffmpeg -i merge.mkv -strict experimental -c copy merge.mp4

Impact: client unable to work with the file

You could say this was a kind of service outage for the client. This would obviously impact the productivity of the person(s) working on the client.

Solution: restart the workstation service on client

The smb connection was not listed with net use, so net use was not the right approach to delete the session/connection for the share in this case. I found a few posts suggesting that a restart of the clients workstation service would clear out sessions/credentials and could solve issues - it did. I have a feeling I've used this approach in the past - it was just too long ago to remember it.

Prior to restarting the service the smb connections list looked like this via elevated PowerShell:

PS C:\WINDOWS\system32> Get-SmbConnection

ServerName ShareName UserName Credential Dialect NumOpens
---------- --------- -------- ---------- ------- --------
omv.blah.local share NODE-5900X\user NODE-5900X\user 3.1.1 10

To restart the workstation service - from an elevated cmd prompt:

net stop workstation && net start workstation

💡 Its important to ensure all explorer and other programs using the share are closed, otherwise this solution might not work as advertised.

Alternative: logging off the windows user and/or restarting the windows node would likely of also resolved this issue. However those approaches are disruptive and sometimes highly undesirable because they can impact peoples workflow and productivity.

Thursday, January 26, 2023

awk - multiple iterations on a single file with different logic (FS) per iteration

Holy smokin’ Toledo’s - what a blast from the past! I was doing some blog spring cleaning and found this unpublished draft from 2012-May-02. Here is the unedited awk paste (scroll down for a commented version):

awk -v loops=2 '
BEGIN {
f = ARGV[ARGC-1]
"wc -l "f"|egrep -o [0-9]+" | getline NL;
while (++i < loops) {
ARGV[ARGC++] = f
}
}
FNR == 1 {
iteration++; print "iteration: "iteration
}
FNR == NL {
FS = "[0-9][0-9]:[0-9][0-9] "
}
iteration == 1 { print $1 }
iteration == 2 { print $NF }

' $media_files_file

In 2012 I must have been happy with my handy work - because I had created a blog draft either as a scratch pad or because I wanted to share it...😊

So I searched my script library for the keyword media_files_file and got a hit on a script named update-dm-to-dc.sh which was created to batch update the date modified timestamp of files to match the date created timestamp.

This was also back in a time when I was probably storing media files on NTFS which stores creation timestamps for its files.

I'm guessing that I probably did something to change the modified timestamp on a bunch of media files and wanted to revert that change. Back in 2012, my storage did not support CoW or a file system that supported snapshots - so there was no easy way to roll back if the mass modification of files went wrong - backups would have been the main undo workflow.

Perhaps iTunes or similar media library had updated file modified timestamps in an undesirable way. E.g. causing differential backup scripts to see the media files as being modified and being selected during the next backup? 

Maybe I was migrating files to a filesystem that didn't support creation timestamps (like XFS v4 or ext3) and wanted to set the modified stamp to the source filesystems creation stamp?

<rabbit-hole>


Q: Did XFS support creation time (crtime) in 2012?
A: No - In 2012 the latest XFS release was v4 - the crtime code (XFS inode v3) was first committed in 2013-Apr-21 by Christoph Hellwig. Here is the commit.
An XFS status update in 2013-May mentions the the release of Linux 3.10 with an "experimental XFS feature - CRC protection for on-disk metadata structures", which AFAIK was part of the inode v3 code.
In 2015-May xfsprogs-3.2.3-rc1 was released which mentioned properly supporting inode v3 format.
The XFS docs were updated in 2016-Jan with details of crtime and XFS v5 fields. Note the differentiation between XFS version and XFS inode version.

Useful related links:
#1 How to find creation date of file?
#2 What file systems on Linux store the creation time? - XFS v5 supports crtime

# on XFS v5
# xfs_db -r -c "inode $(stat -c '%i' yourfile.txt)" -c "print v3.crtime.sec" /dev/disk/by-uuid/1d5722e2-a5ac-XXXX-XXXX-392290480c23
v3.crtime.sec = Sun Aug 22 14:58:40 2021 

It is noted that stat -c '%w' or '%W' should display file creation time on filesystems that store creation time. Noting that on Linux this requires coreutils 8.31 (released 2019-Mar), glibc 2.28 and kernel version 4.11 or newer.

</rabbit-hole>

I've gone through the awk script and added comments - it took me a moment to figure out what I was trying to do back then... Unfortunately I don't have bash history going back to that exact point in time to know the exact contents or line format of $media_files_file.

What I was able to establish is that I probably ran this ad hoc awk script the day before I created the update-dm-to-dc.sh script (the research and debugging phase). So the line format of $media_files_file was probably an earlier iteration of the formats used in the update-dm-to-dc.sh script - the record formats in the script don't match the pattern: FS = "[0-9][0-9]:[0-9][0-9] "

My summary of the ad-hoc awk script: I was trying to run awk once but read the records (lines) more than once and do something different with the the records based on the iteration.

For each record (line in this case - default RS) in the $media_files_file the ad-hoc awk would of:

  1. Once at the start of each iteration - printed the iteration number.
  2. For the #1 iteration - for each record - print the first field (default FS).
  3. For the #2 iteration - for each record - print the number of fields (custom FS).

My hypothesis of the goal of the ad-hoc awk - was to help validate records (lines) and fields - to ensure that the fields were normalised and predictable for script input, with the goal of doing a mass update/change of timestamps of my media library. So this awk was part of the manual assertions to check that the planned script input would work in a predicable way. With comments:

awk -v loops=2 '
# this block is run once at the start of awk invocation
BEGIN {
# store the first awk argument in var f - the file to process in this case
f = ARGV[ARGC-1]

# store the line count of f var in NL var (number of lines in the file to be processed).
# awk does not have a built in variable for this.
"wc -l "f"|egrep -o [0-9]+" | getline NL;

# duplicate the command line argument to to satisfy the number of specified loops.
# this has the effect of telling awk to run more than once on the input file stored in f var.
while (++i < loops) {
ARGV[ARGC++] = f
}
}

# run the following code for each iteration

# this block is executed at the start of each iteration in f var (first line of the file)
FNR == 1 {
# increment and print the iteration counter
iteration++;
print "iteration: "iteration
}

# this block is executed at the end of an iteration in f var (last line of the file)
FNR == NL {
# modify the awk FS (Field Separator)
FS = "[0-9][0-9]:[0-9][0-9] "
}

# this block is executed only during iteration 1
iteration == 1 {
# print the first field in the current input record
print $1
}

# this block is executed only during iteration 2
iteration == 2 {
# print the NF (Number of Fields) in the current input record
print $NF
}

' $media_files_file

Overall thoughts

The script is pretty nifty - in retrospect there may have been better ways to achieve the same results but I like how it explores the possibilities of awk, demonstrating a practical modification of ARGV and how to run different logic on the same input records. Nice work 2012 me!