Skip to content

trims zsv related metadata from column names - #7963

Open
jangorecki wants to merge 1 commit into
masterfrom
zsv
Open

jangorecki wants to merge 1 commit into
masterfrom
zsv

Conversation

@jangorecki

@jangorecki jangorecki commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Background: it is possible to push limits of fread when reading from pre-indexed csv.

Reproducible example is taken from: Filter big csv early by partitions in zsv/README.md

stopifnot(nchar(system("zsv --version", intern=TRUE)) > 8L) # ensure zsv installed

library(data.table)
make_data = function(n = 1e6, years) {
  fwrite(data.frame(
    id = sort(sample(1.05*n, n)), # 5pct missing
    year = sort(sample(years, n, TRUE)),
    amount = sample(25, n, TRUE)
  ), "data.csv")
  # zsv is csv rows + zsv format header row.
  system("zsv data.csv --detect-types --index=id --partition-by=year > data.zsv")
  invisible(TRUE)
}
n = 1e9 # requires more than 25GiB, 17G csv on disk
make_data(n, years = 2020:2026)
system("head -3 data.csv")
system("head -3 data.zsv")

system.time(
  a1 <- fread("data.csv")[year %in% 2021:2023]
)
#   user  system elapsed
# 26.022   7.229   6.111
system.time(
  a2 <- fread("zsv data.zsv --partition=2021..2023")
)
#   user  system elapsed
#  8.887   3.503   3.306
stopifnot(all.equal(a1, a2))

make_data(n, years = 1970:2026)
system.time(
  a1 <- fread("data.csv")[year %in% 2021:2023]
)
#   user  system elapsed
# 27.542   5.777   5.597
system.time(
  a2 <- fread("zsv data.zsv --partition=2021..2023")
)
#   user  system elapsed
#  1.111   0.489   0.475
stopifnot(all.equal(a1, a2))

zsv internally is still csv, just longer column names.


This PR trims zsv related metadata from column names when reading such csv directly via fread("tiny.zsv"):

cat > tiny.csv << EOF
id,year,city,amount
1,2024,Bern,2.5
2,2024,Basel,
4,2025,Zug,1.5
5,2026,Bern,3
EOF
cat > tiny.zsv << EOF
id~u8-index~0~zsv,year~u16-partition{2024@0|2025@30|2026@45},city~str,amount~f64
1,2024,Bern,2.5
2,2024,Basel,
4,2025,Zug,1.5
5,2026,Bern,3
EOF
library(data.table)

# master
fread("tiny.csv")[1]
#      id  year   city amount
#1:     1  2024   Bern    2.5
fread("tiny.zsv")[1]
#   id~u8-index~0~zsv year~u16-partition{2024@0|2025@30|2026@45} city~str amount~f64
#1:                 1                                       2024     Bern        2.5

# branch
fread("tiny.csv")[1]
#      id  year   city amount
#1:     1  2024   Bern    2.5
fread("tiny.zsv")[1] # verbose: "Detected zsv header format: trim zsv to csv"
#      id  year   city amount
#1:     1  2024   Bern    2.5

@github-actions

Copy link
Copy Markdown
  • HEAD=zsv much slower for transform improved in #5493
  • HEAD=zsv faster P<0.001 for memrecycle regression fixed in #5463
  • HEAD=zsv faster P<0.001 for DT[by] max regression fixed in #7480

(Plot image upload to picrd.com failed; download the artifact below to view plots)

Generated via commit 19108f2

Download link for the artifact containing the test results: ↓ atime-results.zip

Task Duration
R setup and installing dependencies 1 minutes and 34 seconds
Installing different package versions 11 minutes and 51 seconds
Running and plotting the test cases 4 minutes and 53 seconds

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant