-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Objects show as "updated" and are redownloaded even when unchanged #53
Comments
Thank you for the very detailed report. Can you please run the following code and paste the outputs of the two calls here: import { S3SyncClient, ListLocalObjectsCommand, ListBucketObjectsCommand } from 's3-sync-client';
const client = new S3SyncClient({ /* your config */ });
console.log(
await client.send(
new ListLocalObjectsCommand({
directory: 'output',
})
)
);
console.log(
await client.send(
new ListBucketObjectsCommand({
bucket: 'my-bucket',
prefix: 'path/to/directory',
})
)
); The diff code for updates is pretty simple: if (
sourceObject.size !== targetObject.size ||
(options?.sizeOnly !== true &&
sourceObject.lastModified > targetObject.lastModified)
) {
updated.push(sourceObject);
} Let's see if the issue comes from values or maybe value types. |
Sure thing, here's the local object output, truncated for brevity: [
LocalObject {
id: 'test-obj-a.jpg',
size: 1667804,
lastModified: 1685735624000,
isExcluded: false,
path: 'output/test-obj-a.jpg'
},
LocalObject {
id: 'test-obj.b.jpg',
size: 385869,
lastModified: 1685735634000,
isExcluded: false,
path: 'output/test-obj.b.jpg'
}
] and the bucket object output: [
BucketObject {
id: 'test/test-obj-a.jpg',
size: 1667804,
lastModified: 1685735624935,
isExcluded: false,
bucket: '...',
key: 'test/test-obj-a.jpg'
},
BucketObject {
id: 'test/test-obj-b.jpg',
size: 385869,
lastModified: 1685735766762,
isExcluded: false,
bucket: '...',
key: 'test/test-obj-b.jpg'
}
] Looks like the |
I've made tests on S3, and it seems that AWS doesn't store milliseconds for the LastModified field. Ref: aws/aws-cli#5369 My test with official AWS SDK commands: await s3Client.send(
new PutObjectCommand({
Bucket: BUCKET_2,
Key: 'def/jkl/xmoj',
Body: Buffer.from('0x1234', 'hex'),
})
);
console.log(
(
await s3Client.send(
new ListObjectsV2Command({
Bucket: BUCKET_2,
Prefix: 'def/jkl/xmoj',
})
)
).Contents.map(({ LastModified }) => LastModified.getTime())
);
// => [ 1685740748000 ]
console.log(
(
await s3Client.send(
new GetObjectCommand({
Bucket: BUCKET_2,
Key: 'def/jkl/xmoj',
})
)
).LastModified.getTime()
);
// => 1685740748000 Can you run the last two commands on |
You are correct, that is the case. Here's the output: console.log(
(
await clientS3.send(
new ListObjectsV2Command({
Bucket: env.S3_BUCKET,
Prefix: 'test/test-obj-a.jpg',
})
)
).Contents.map(({ LastModified }) => LastModified.getTime())
);
// => [ 1685735624935 ]
console.log(
(
await clientS3.send(
new GetObjectCommand({
Bucket: env.S3_BUCKET,
Key: 'test/test-obj-a.jpg',
})
)
).LastModified.getTime()
);
// => 1685735624000 Is there anything that can be done to work around that by disregarding the milliseconds if they are returned? |
I'm not sure we can safely round or truncate values. If you look at I would suggest opening a ticket with the providers and in the meantime using the |
Whoops! That was a mistake on my part, I think I changed something in #53 (comment) that caused the times to shift ( [
LocalObject {
id: 'test-obj-a.jpg',
size: 1667804,
lastModified: 1685735624000,
isExcluded: false,
path: 'output/test-obj-a.jpg'
},
LocalObject {
id: 'test-obj-b.jpg',
size: 385869,
lastModified: 1685735766000,
isExcluded: false,
path: 'output/test-obj-b.jpg'
}
]
[
BucketObject {
id: 'test/test-obj-a.jpg',
size: 1667804,
lastModified: 1685735624935,
isExcluded: false,
bucket: 'my-bucket',
key: 'test/test-obj-a.jpg'
},
BucketObject {
id: 'test/test-obj-b.jpg',
size: 385869,
lastModified: 1685735766762,
isExcluded: false,
bucket: 'my-bucket',
key: 'test/test-obj-b.jpg'
}
] As you can see, the timestamps for each object are exactly the same apart from the milliseconds, so it doesn't appear to be an issue with the provider. |
Environment
NodeJS v18.16.0
macOS 13.4 Ventura
Steps to reproduce
Sample code for how I'm invoking
s3-sync-client
(with sensitive values stripped):Show
Expected result
Items that are unchanged from the remote to the local system should not be recopied.
Actual result
Even after an initial successful sync to the local filesystem,
s3-sync-client
continues to redownload files that haven't changed, incurring additional bandwidth charges.Here's a screen capture of the network requests going across:
Show
And the resulting output:
Show
Other items of note:
s3-sync-client
when performing the diff is the same time as what's on the filesystem and on the WebUI for the S3 storage services so any drift differences, if they exist, aren't visible at least to my eyes as an end-user.Happy to provide any other relevant details!
The text was updated successfully, but these errors were encountered: