Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Memos for Visual FoxPro #118

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

d4mation
Copy link

@d4mation d4mation commented Sep 19, 2022

Changes were made specifically for Visual FoxPro (TableType::VISUAL_FOXPRO), but it may work for other Versions too.

There are some minor issues with this PR, but they would likely require some refactoring of the library to handle nicely.

I've described the lingering issues below


Firstly, Visual FoxPro only expects 4 bytes for Memo fields, but XBase\Header\Column\Validator\DBase\MemoValidator will force this to 10 bytes no matter how you define the Column.

class MemoValidator implements ColumnValidatorInterface
{
const LENGTH = 10;
public function getType(): array
{
return [
FieldType::MEMO,
];
}
public function validate(Column $column): void
{
$column->length = self::LENGTH;
}
}

As a result, the Record Byte Length for any Tables that include Memo Fields will end up being incorrect. This will also impact the offset of the Column within the Record for any Fields that come after a Memo.

Given the way that the Memo field validation is being handled "globally" I'm not sure if there is a particularly good way to handle a different byte length only for specific DBase versions.

Secondly, while the .FPT file is written, technically the .DBF file has no knowledge that it exists. The Table Flag will need to be set accordingly in the .DBF file's header if Memo fields exist.

I've corrected these two issues on my end by tweaking the .DBF file after initial creation with the following code:

// Example Columns. These Columns were used higher up in my code to create the Table similar to the examples in the README.
// README Example: https://github.com/luads/php-xbase/blob/e818e3526102a20b7d9c7e1b14feef686c973a5d/README.md?plain=1#L169-L198
$columns = array(
    array(
        "name" => "TESTCHAR",
        "type" => FieldType::CHAR,
        "length" => 10,
    ),
    array(
        "name" => "TESTMEMO",
        "type" => FieldType::MEMO,
        "length" => 4,
    ),
    array(
        "name" => "ANOTHERCHAR",
        "type" => FieldType::CHAR,
        "length" => 15,
    )
);

$memo_columns = array_filter( $columns, function( $column ) {
    return $column['type'] == FieldType::MEMO;
} );

if ( ! empty( $memo_columns ) ) {

    $file_stream = Stream::createFromFile( $path, 'rb+' );

    // The Table flag will need to be set so other applications will attempt to load the Memo file
    $file_stream->seek( 28 );
    $file_stream->writeUChar( 2 );

    // Most DBase files use a length of 10 bytes for Memo fields, but Visual FoxPro uses 4 bytes
    // We are going to subtract 6 bytes from the Record Length for each Memo field
    // https://web.archive.org/web/20210801135633/http://devzone.advantagedatabase.com/dz/webhelp/advantage9.0/server1/dbf_field_types_and_specifications.htm

    $memo_field_count = count( $memo_columns );

    // Record Length is stored in bytes 10-11
    $file_stream->seek( 10 );
    $saved_record_length = $file_stream->readUShort();

    // Write the corrected Record Length to the proper offset
    $file_stream->seek( 10 );
    $file_stream->writeUShort( $saved_record_length - ( $memo_field_count * 6 ) );

    // We need to patch the length and offset of each Column individually too
    
    $removed_bytes = 0;
    foreach ( $headers as $index => $column ) {

        // Columns start at byte 32 and are 32 bytes long themselves
        $column_header_offset = 32 * ( $index + 1 );

        if ( $removed_bytes > 0 ) {

            // The offset of the column in the record is stored in bytes 12-15 of the column
            $file_stream->seek( $column_header_offset + 12 );
            $old_record_column_offset = $file_stream->readUShort();

            // Write the offset with the removed bytes subtracted from it, effectively bumping it backward the correct amount
            $file_stream->seek( $column_header_offset + 12 );
            $file_stream->writeUShort( $old_record_column_offset - $removed_bytes );

        } 

        if ( $column['type'] == FieldType::MEMO ) {

            // Write the correct Column length, stored in byte 16
            $file_stream->seek( $column_header_offset + 16 );
            $file_stream->writeUChar( 4 );

            // 10 bytes minus 4 bytes is 6 bytes. Increment our "Removed bytes" to subtract from subsequent columns
            $removed_bytes += 6;

        }

    }
    
    $file_stream->close();

    // This helps to confirm that our adjusted Header worked by attempting to parse the Header
    $table_reader = new TableReader( $path );
    $table_reader->close();

}

Lastly, one thing that may need to be tweaked is that the Memo file is created based on whether the current Table version supports Memos rather than if any Memo Columns exist or not. Currently, every time the Table is saved (meaning every time a Record is added) it will attempt to create a Memo file even if one isn't needed. I'm not sure if there's a good way around this though, since I know the parser can be set to only load certain Columns. If the Columns were restricted like this, it may not see the Memo columns in order to make this decision.

Changes were made specifically for Visual FoxPro
(TableType::VISUAL_FOXPRO), but it may work for other Versions too
@@ -25,7 +26,8 @@ public static function create(Table $table, EncoderInterface $encoder): ?MemoInt
}
$memoFilepath = $fileInfo['dirname'].DIRECTORY_SEPARATOR.$fileInfo['filename'].$memoExt;
if (!file_exists($memoFilepath)) {
return null; //todo create file?
$memo_creator = MemoCreatorFactory::create($table);
Copy link
Collaborator

@gam6itko gam6itko Sep 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the DB does not have memo fields, then a memo file will still be created. It is not right. TableWriter should be responsible for creating files. MemoFactory is part of TableReader. If memo file missing (but it should be), we need to pass this info to MemoFactory of use something like fixer-tool

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, you're right. Without that line changed in MemoFactory it is creating the Memo file. I'm not sure why I thought changing that was necessary 🤔

Anyway, currently TableWriter appears to always create an empty Memo file if the TableType supports Memos.

TableEditor has a nice method for grabbing any set Memo Columns, but TableWriter is its own Class that doesn't have access to TableEditor's methods so we cannot currently use that. Would it be appropriate to move that method to a Trait so that both TableEditor and TableWriter can use it?

If TableWriter had that method, we could have it check whether or not the Memo file needs to be created instead of always creating it.

Copy link
Collaborator

@gam6itko gam6itko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Yes, you are right, the library needs another refactoring. The problem with FoxPro is that I do not work with it and do not know all the subtleties.

There are not enough test cases in your PR. Can you write some?

@@ -22,6 +22,8 @@ public static function create(Table $table)
case TableType::DBASE_7_MEMO:
return new DBase7MemoCreator($table);
//todo foxpro
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we delete this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The todo? That I'm not sure, actually. I don't know anything about the other FoxPro versions and I've found that documentation is a bit scarce. I'm not sure if my changes here will work properly for the other versions :/

src/Memo/Creator/FoxProMemoCreator.php Outdated Show resolved Hide resolved
@d4mation
Copy link
Author

There are not enough test cases in your PR. Can you write some?

What tests specifically would you like to see? I'd be happy to write some, but without the refactoring necessary to be able to handle the Memo Field length, the Table Flag, and the issue I discovered in #117 I don't know how best to confirm that the file generation is occurring properly.

TableWriter is handling creation of the Memo file, so it doesn't need to
be done here.

Referencing luads#118
The Backlist is meant to link to a Database (.DBC) file. Some
applications (FoxPro specifically, I've noticed) get confused when it is initialized with non-0 values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants