Uncovering the Past: Reconstructing File Activity from Ext4 and XFS Journals

Uncovering the Past: Reconstructing File Activity from Ext4 and XFS Journals

Speakers: Minoru Kobayashi, Forensic Investigator and CSIRT Member, Internet Initiative Japan Inc.

Conference: CODE BLUE

YouTube: https://www.youtube.com/watch?v=TZtAnawtlNk

Overview

In the realm of digital forensics, uncovering a complete and accurate timeline of file system activity is paramount to understanding the scope and nature of a security incident. However, traditional methods often fall short, particularly in Linux environments. Minoru Kobayashi's talk, "Uncovering the Past: Reconstructing File Activity from Ext4 and XFS Journals," presented at CODE BLUE, addresses this critical gap. Kobayashi, a seasoned forensic investigator and CSIRT member at Internet Initiative Japan Inc., delves into the largely untapped potential of Ext4 and XFS file system journals as rich sources of historical file activity.

The presentation outlines a novel approach to parse and interpret these journals, which are typically designed for crash recovery, not forensic analysis. By meticulously analyzing the low-level metadata changes recorded within these journals, Kobayashi demonstrates how to reconstruct detailed file activity timelines, revealing crucial actions that might otherwise remain hidden. He introduces FJTA (Forensic Journal Timeline Analyzer), a new open-source tool developed to automate this complex process. This research is particularly significant for incident responders and forensic analysts working with Linux systems, as it provides a powerful new avenue for evidence collection, capable of exposing sophisticated attacker techniques like timestamp tampering and hidden persistence.

Background

▶ Watch: Introduction: Why Linux file system journal forensics? (0:00)

The foundation of forensic timeline analysis often relies on MacB (Modification, Access, Creation, Birth) timestamps. While these timestamps provide a quick overview of file activity, they suffer from significant limitations. Primarily, MacB timestamps only record the last instance of an event, providing no historical context. More critically, they are highly vulnerable to manipulation; attackers can easily alter these timestamps using tools like touch or more sophisticated methods, effectively obfuscating their actions. This fragility means that relying solely on MacB times can lead to incomplete or misleading investigative outcomes, potentially allowing malicious activity to go unnoticed.

In contrast to Linux, the Windows ecosystem offers more robust journaling capabilities for forensic purposes, notably the NTFS $LogFile. Specialized tools exist to parse this journal, allowing investigators to reconstruct a detailed history of file system operations. For Linux, however, a comparable, widely adopted solution for Ext4 and XFS journals has been conspicuously absent. These file systems, being the default for many popular Linux distributions (Ext4 for Debian/Ubuntu, XFS for Red Hat Enterprise Linux), are ubiquitous in server and desktop environments alike.

Existing Linux forensic tools like TSK (The Sleuth Kit), while powerful for file system analysis, offer only limited support for journals. TSK's jls command can list journal data, and jcat can dump specific journal blocks, but neither provides the high-level interpretation necessary to reconstruct file operations. Furthermore, the latest versions of TSK still lack XFS journal support. While some proof-of-concept scripts have been published over the years, they often require prior knowledge of specific file paths or inode numbers and are not practical for comprehensive, large-scale investigations. This long-standing gap in Linux forensic capabilities motivated Minoru Kobayashi's research, aiming to unlock the full potential of Ext4 and XFS journals as invaluable, tamper-resistant forensic artifacts.

Key Findings

▶ Watch: Methodology: Inferring file activity from transaction changes (16:10)

Minoru Kobayashi's research yields several pivotal findings that significantly advance the state of Linux file system forensics:

1. Rich, Low-Level Activity Data: Both Ext4 and XFS journals, despite their primary purpose of crash recovery, store detailed, low-level records of metadata changes. This includes changes to file attributes, directory entries, and inode structures, providing a granular history of operations far beyond what MacB timestamps offer.
2. Reconstructible Timelines: By meticulously parsing and interpreting the structured data within these journals, it is possible to reconstruct a comprehensive timeline of file system activities. This timeline reveals not just the last action, but a sequence of events, including file creation, modification, deletion, permission changes, and even access patterns.
3. Inference of File Activity: A key discovery is the ability to infer specific file activities by comparing consecutive journal transactions. By observing changes in inode fields (e.g., UID, permissions, timestamps) or the appearance/disappearance of inodes across transactions, actions like SUID changes, timestamp tampering, and file deletions can be accurately identified.
4. Automated Analysis with FJTA: The development of FJTA (Forensic Journal Timeline Analyzer) demonstrates the practical application of these findings. This tool effectively automates the complex parsing, interpretation, and timeline generation process for both Ext4 and XFS journals, making advanced journal forensics accessible to a wider audience.
5. Uncovering Hidden Malicious Activity: The research proves that FJTA can uncover sophisticated attacker techniques that evade traditional forensic methods. This includes detecting timestamp manipulation (timestamp "stomping") and identifying persistence mechanisms that rely on modifying file attributes or creating hidden artifacts, even when MacB times have been altered.
6. Resilience Against Anti-Forensics: Attempts by attackers to delete or overwrite file system journals using standard system utilities or direct dd commands were shown to be largely ineffective. Such actions either fail on mounted file systems or render the system unbootable, making journals a relatively robust and tamper-resistant source of evidence. While flooding the journal with legitimate operations can overwrite old entries, this itself creates a detectable pattern of high activity.

These findings collectively establish file system journal analysis as a critical and indispensable component of a comprehensive Linux incident response strategy, providing a deeper, more accurate understanding of system compromise.

Technical Deep Dive

▶ Watch: Demo: FJTA finds suspicious files TSK missed (23:00)

The core of Minoru Kobayashi's work lies in the intricate parsing and interpretation of Ext4 and XFS journal structures. While both serve the same purpose of crash recovery, their internal layouts and data representations differ significantly, necessitating distinct parsing approaches.

Ext4 Journal Structure and Parsing

The Ext4 journal is a crucial component that ensures file system consistency. Its location is determined by the file system's superblock, typically found at offset 0x400 from the partition start. Within this superblock, the inode number of the journal file is stored, usually defaulting to 8. To access the journal data, one must traverse the block group descriptors to locate the inode table, then parse the 8th inode's structure to find the journal's physical block numbers.

The Ext4 journal operates on transaction units, each designed to record a set of related metadata changes. A transaction begins with a Descriptor Block, followed by one or more Data Blocks, and concludes with a Commit Block. This sequence represents a single atomic operation. A critical characteristic of the Ext4 journal is its circular buffer nature; older entries are continuously overwritten by newer ones, making direct data carving challenging.

Key structures within an Ext4 journal transaction include:

• Journal Superblock: Contains metadata about the journal itself, such as its size and block numbers.
• Descriptor Block: Acts as a header for the transaction, primarily containing block tags.
• Block Tags: These are crucial for reconstruction. Each tag corresponds to a data block in the transaction and contains t_block_nr and t_block_nr_high fields, which specify the original, intended block number on the file system where the data should be written. This information is vital for recovery and, forensically, for understanding the context of the data.
• Data Blocks: These blocks contain the actual metadata (e.g., inode table entries, directory entries) that are being modified. A significant challenge in parsing Ext4 data blocks is the lack of magic numbers or clear type identifiers within the blocks themselves. For example, an inode table entry or a directory entry might not have a distinct signature, making it difficult to determine its type purely from its content.
• Commit Block: Marks the successful completion of a transaction and contains a timestamp, which is valuable for forensic timeline reconstruction and detecting timestamp tampering.

FJTA's parsing strategy for Ext4 addresses the data block identification challenge. It leverages the block tags: if a data block's t_block_nr corresponds to an inode table block, it can be confidently parsed as an inode. For other data blocks, FJTA employs a heuristic approach, attempting to parse them with various known structures (e.g., directory entries) until a valid interpretation is found.

XFS Journal Structure and Parsing

XFS, designed for high-performance file systems, also employs a journal, but with a more structured and self-describing format. The XFS journal's physical location parameters are stored directly within the file system's superblock at offset 0.

The XFS journal is organized into log records. Each log record comprises a Log Record Header, followed by one or more Log Operation Headers, and then corresponding Log Items. Unlike Ext4, XFS journals record data in host byte order, meaning parsers must handle both little-endian and big-endian architectures, as indicated by fields within the log record header. Like Ext4, the XFS journal is also a circular buffer.

Understanding the XFS inode structure is helpful for journal analysis:

• XFS Inode: Divided into three main fields: Inode Core (basic metadata like file type, timestamps), Data Fork (stores directory entries or symlink targets), and Attribute Fork (for extended attributes). The XFS inode is identifiable by a magic number, IN.

Key structures within an XFS log record include:

• Log Record Header: Contains the total length of the log record, the number of log operations within it, and a log_format field. A log_format value of 1 indicates that the log items are recorded in little-endian.
• Log Operation Header: Contains flags indicating the start or end of a transaction, and the length of the associated log data. Transactions can span multiple log records.
• Log Item: These are highly structured and typically begin with a magic number that identifies the type of operation (e.g., inode update, directory entry change). They also contain a i_not_fields bitmap, which explicitly indicates which parts of the inode (core, data fork, attribute fork) are being updated. If a field is not updated, it is not recorded in the log item.

FJTA's parsing of XFS journals benefits from this inherent structure. The explicit magic numbers and flags within the log items and headers allow for more direct and reliable determination of data types and operations, simplifying the parsing logic compared to Ext4.

File Activity Inference and Timeline Construction

Regardless of the file system, FJTA employs a common logic for inferring file activity by comparing sequential journal transactions (N and N+1). This comparative analysis allows for the detection of various events:

• Permission/Ownership Changes: If the UID, GID, or permission bits of an inode change between N and N+1, it indicates a chmod or chown operation (e.g., SUID bit setting).
• Timestamp Tampering: A rewind in the Mtime (modification time) or CRtime (creation time) suggests timestamp manipulation. The commit block timestamp can help validate the true event time.
• Deletion: If an inode's Dtime (deletion time) is set, or if an inode present in transaction N is absent in N+1, it indicates a file deletion.
• Creation: The appearance of a new inode in N+1 that was not present in N, especially if its timestamps align, indicates a file creation.

Once these activities are inferred, FJTA constructs a timeline by ordering them by their transaction commit times. A crucial aspect is linking inode numbers to filenames. Since filenames are stored in directory entries, and a single inode can have multiple hard links (i.e., multiple filenames), FJTA records all associated filenames and their parent directory inode numbers for clarity and disambiguation in its NDJSON output. This allows for a comprehensive, human-readable record of file system events.

FJTA's architecture standardizes the parsed data from both Ext4 and XFS into a common internal format, facilitating consistent activity inference and timeline generation. It supports various disk image formats, including E01, raw, and VMDK, providing flexibility for forensic acquisition.

Demo / Proof of Concept

▶ Watch: Demo: Detecting timeline tampering and time stomping (28:00)

Minoru Kobayashi's presentation included a compelling demonstration of FJTA's capabilities, showcasing its ability to uncover hidden malicious activity that traditional tools might miss. The scenario involved a simulated system compromise where an attacker had attempted to cover their tracks.

The premise was an incident detected by a CM (Centralized Management) system on October 9, 2025, at 04:47:28. The forensic investigation began by examining file activity from 04:35 onwards. An initial analysis using TSK's timeline generation tools yielded no suspicious filenames or activities, suggesting the attacker had successfully hidden their actions.

However, when FJTA was deployed, a very different picture emerged. The tool, after parsing the Ext4 journal, revealed several highly suspicious files that were not apparent in the TSK output:

• panixshell
• excstomp
• preload_backdoor.so
• lds_preload.so

A detailed timeline reconstructed by FJTA exposed the following sequence of events:

1. File Creation and Execution: At 04:38:28, panixshell and excstomp were created. Shortly after, their permissions were changed to 775 (setting the execute bit) via a chmod operation. Subsequent access events for these files strongly implied their execution, indicating the attacker ran these scripts or binaries.
2. Deletion and Inode Reuse: The timeline showed panixshell and excstomp were later deleted. FJTA inferred this even without an explicit "delete" log entry. By observing that the inode associated with panixshell was reused for a new file with a different CRtime (creation time), FJTA correctly deduced that the original file was deleted, and its inode was quickly reallocated. This is a crucial distinction, as a simple rename operation would not alter the CRtime.
3. Timestamp Tampering (Anti-Forensics): Most critically, FJTA uncovered evidence of timestamp tampering on preload_backdoor.so and lds_preload.so. These files, initially created around 04:38, had their CRtime and Mtime rewound to October 10, 2020, at 01:10. This is a classic anti-forensic technique, often used for persistence mechanisms like LD_PRELOAD hooks, where attackers modify timestamps to make their files appear older and less suspicious. FJTA explicitly tagged these events as "timestamp stomping."

The demo conclusively illustrated that FJTA could provide granular insights into attacker actions, including file creation, permission changes, execution, deletion, and sophisticated timestamp manipulation. This information was entirely absent from the TSK-generated timeline, highlighting FJTA's unique value in uncovering critical forensic evidence.

Defensive Implications

▶ Watch: Q&A: Motivation for tool and Systemd Journal difference (36:30)

Minoru Kobayashi's research and the FJTA tool provide invaluable insights for defenders, but also underscore critical limitations and considerations for journal forensics.

Limitations of Journal Forensics

While powerful, file system journals were primarily designed for crash recovery, not comprehensive forensic logging, leading to inherent limitations:

• Limited Storage and Circular Nature: Journals are allocated a finite amount of storage (e.g., Ext4 up to 1GB, XFS up to 2GB, compared to NTFS $LogFile at 64MB, which is smaller but often more robustly managed for forensics). Due to their circular buffer design, older entries are continuously overwritten by new activity. This means that on busy partitions (like /var/log), journal data might be retained for only a short period, making early detection and rapid response crucial.
• Difficulty in Full Path Reconstruction: Journals typically only record changes to directory entries, not the full hierarchical path of a file. For files created during OS installation or those in stable directories (e.g., /), there might be no journal entries for their parent directories. Therefore, reconstructing the full path of an inode from journal data alone is often impossible. FJTA mitigates this by linking inode numbers to filenames and their parent directory inode IDs, offering partial context.
• Carving Challenges: The circular, non-contiguous nature of journal data makes it extremely difficult to carve meaningful information from a raw disk image without proper parsing of the journal's structure.

Anti-Forensic Considerations

Kobayashi also explored common anti-forensic attempts to neutralize journal evidence:

1. Deleting the Journal:

• Ext4: tune2fs -O ^has_journal
• XFS: xfs_repair -L
• Outcome: These commands cannot be executed on a mounted file system. Thus, an attacker cannot easily delete the journal on a live, running system without first unmounting it, which would cause significant disruption and likely trigger alerts. This method is generally ineffective for stealthy attackers.

1. Overwriting the Journal with dd:

• Method: Directly overwrite the journal's allocated blocks with null bytes using dd.
• Outcome: For both Ext4 and XFS, attempting to dd overwrite the journal (even excluding the superblock) resulted in system instability, hangs, and ultimately an unbootable state. The file system integrity checks would fail upon reboot. This renders the system unusable, making it an impractical anti-forensic technique for an attacker aiming for stealth or continued access.

1. Flooding the Journal with Normal Operations:

• Method: Performing a large volume of legitimate file system operations (e.g., creating, modifying, deleting numerous temporary files) to intentionally overwrite older journal entries.
• Outcome: This method is theoretically effective in overwriting old journal data. However, it generates a massive amount of file system activity.
• Defense: This high volume of activity itself serves as a strong indicator of compromise. Audit logs, syslog, or other system monitoring tools would likely record this unusual burst of file operations. Furthermore, FJTA itself would capture this anomaly, showing an abnormally high number of file creations/modifications within a short timeframe.

Key Defensive Actions

• Integrate Journal Analysis: Defenders should incorporate tools like FJTA into their incident response playbooks. Journal analysis complements traditional timeline analysis and log aggregation, providing a deeper forensic capability for Linux systems.
• Prioritize Early Detection: Due to the limited retention period of journals, swift detection of incidents is crucial. The sooner an investigation begins, the higher the chance of recovering valuable journal data.
• Monitor for Anomalous Activity: Look for unusual patterns such as attempts to unmount critical file systems, high rates of file creation/deletion, or suspicious dd commands, which might indicate anti-forensic efforts.
• Preserve Evidence: In the event of a suspected compromise, creating a full disk image as quickly as possible is paramount to preserving all forensic artifacts, including the file system journal.
• Understand Persistence Mechanisms: Be aware that techniques like LD_PRELOAD combined with timestamp tampering are common. Journal analysis is a powerful way to detect these.

Key Takeaways

• Linux file system journals (Ext4, XFS) are rich, tamper-resistant forensic artifacts that record low-level metadata changes, offering a detailed history of file activity beyond what traditional MacB timestamps provide.
• The FJTA (Forensic Journal Timeline Analyzer) tool significantly enhances Linux incident response capabilities by automating the complex parsing and interpretation of these journals, allowing for detailed timeline reconstruction.
• FJTA can uncover hidden malicious activity, including sophisticated timestamp tampering and persistence mechanisms that evade detection by standard forensic methods or when attackers have altered MacB times.
• Anti-forensic attempts to delete or overwrite file system journals are generally ineffective or destructive to the system, making journals a relatively robust source of evidence.
• Despite their forensic value, journal size limitations and circular buffer designs necessitate early detection and rapid response to maximize the chances of recovering critical historical data.
• Integrating journal analysis into incident response workflows is crucial for comprehensive Linux forensics, complementing existing tools and log analysis to gain a complete picture of a compromise.

About the Speaker(s)

Minoru Kobayashi is a Forensic Investigator and CSIRT Member at Internet Initiative Japan Inc. (IIJ). His work primarily focuses on forensic technical investigations and incident response within the company. With a background in tackling various security incidents, Kobayashi brings practical, real-world experience to his research. He has previously shared his expertise at other conferences, demonstrating a commitment to advancing the field of digital forensics. His research into Linux file system journals highlights his dedication to developing innovative solutions for complex investigative challenges.