Fast and efficient Golang package for splitting large csv files on smaller chunks by size in bytes.
- Super-fast splitting. Splitting of 700MB+ file takes less than 1 sec!
- Allocates minimum memory regardless file size.
- Supports multiline cells and headers (csv should follow the basic rules https://en.wikipedia.org/wiki/Comma-separated_values).
- Configurable destination folder.
- Disabling/enabling of copying a header in chunk files.
Install:
go get -u github.com/tolik505/split-csv
Import:
import splitCsv "github.com/tolik505/split-csv"
func ExampleSplitCsv() {
splitter := splitCsv.New()
splitter.Separator = ";" // "," is by default
splitter.FileChunkSize = 100000000 //in bytes (100MB)
result, _ := splitter.Split("testdata/test.csv", "testdata/")
fmt.Println(result)
// Output: [testdata/test_1.csv testdata/test_2.csv testdata/test_3.csv]
}
If copying of a header in chunks is not needed then:
func ExampleSplitCsv() {
splitter := splitCsv.New()
splitter.Separator = ";" // "," is by default
splitter.FileChunkSize = 20000000 //in bytes (20MB)
s.WithHeader = false //copying of header in chunks is disabled
result, _ := splitter.Split("testdata/test.csv", "testdata/")
fmt.Println(result)
// Output: [testdata/test_1.csv testdata/test_2.csv testdata/test_3.csv]
}