Added a range option to `Worksheet.get_notes` [Issue #1482] #1487

muddi900 · 2024-06-24T00:31:11Z

alifeee · 2024-06-25T12:50:08Z

hi! thanks for the change! your code was all good, but the test cassette was breaking things. This is common, and what I usually do is completely delete the JSON test cassette and re-create it, effectively running:

rm tests\cassettes\WorksheetTest.test_get_notes.json
GS_CREDS_FILENAME="tests/creds.json"
GS_RECORD_MODE="all"
pytest -k "test_get_notes" -v -s tests/ 
# check it worked
GS_RECORD_MODE="none"
pytest -k "test_get_notes" -v -s tests/

I have reformatted the test a little so it is easier to read for me, and changed range to grid_range, as range is a python built-in keyword that we should not change.

Also, in testing, I noticed that get_notes did not work with worksheets that were not the primary sheet, so I have fixed this (see 16429af and 983dcb5 above) too.

again thanks for the change! I will let @lavigne958 read this and merge when happy with it :)

nbwzx · 2024-06-29T09:12:38Z

Also, in testing, I noticed that get_notes did not work with worksheets that were not the primary sheet, so I have fixed this (see 16429af and 983dcb5 above) too.

Wait is that wrong?

For example, there are three worksheets (Sheet1, Sheet2, Sheet3).

res["sheets"][self.index]["data"][0].get("rowData", [{}]) returns the notes of current worksheet.
self.spreadsheet.worksheets()[0].get_notes() get the notes of Sheet1
self.spreadsheet.worksheets()[1].get_notes() get the notes of Sheet2
self.spreadsheet.worksheets()[2].get_notes() get the notes of Sheet3

If you change self.index to 0, you can only get the notes of the primary sheet (Sheet1),
self.spreadsheet.worksheets()[0].get_notes() get the notes of Sheet1
self.spreadsheet.worksheets()[1].get_notes() get the notes of Sheet1
self.spreadsheet.worksheets()[2].get_notes() get the notes of Sheet1
how can you get the notes of other worksheets (Sheet2, Sheet3)?

alifeee · 2024-06-29T15:14:37Z

Wait is that wrong?

As it turns out... yes and no! ;]

When you provide a range, i.e., here

https://github.com/muddi900/gspread/blob/d4a57746816ad097d56969cfab4c9c12a483dff6/gspread/worksheet.py#L2657-L2658

Then the returned results has only one sheet. Printing res, we get

> get_notes()
# res = 
{'sheets':
  [
    {'data': [{}]},
    {'data': [{'rowData': [{'values': [{'note': 'the first time'}]}, {}, {'values': [{}, {'note': 'two sheets'}]}]}]}
  ]
}
> get_notes("A2:C3")
# res = 
{'sheets':
  [
    {'data': [{'rowData': [{}, {'values': [{}, {'note': 'two sheets'}]}]}]}
  ]
}

So in the initial case, we do need to index it with worksheet.index, i.e, 1

In the second case, we need to use 0. I had spotted the latter case, which is why I made my change. However, I didn't make a very good test to be able to spot the reason it had changed.

I have made the test clearer (split it out) and 'fixed' it by always providing worksheet.title, so we can only ever expect one sheet in the response.

Thanks for spotting @nbwzx ! :)

alifeee · 2024-06-29T15:17:18Z

sorry for jumping on your PR @muddi900. wouldn't have happened without you ;] so thanks for proposing the change!

will let @lavigne958 review and with an approval I will merge, and this can be included in 6.2.0 :)

muddi900 · 2024-06-30T17:15:35Z

No need to apologize. Thank you for the help.

nbwzx · 2024-07-01T05:28:34Z

@alifeee
I tried your code on one of my sheets. The code get_notes() raises an error: gspread.exceptions.APIError: APIError: [400]: Range (Introduction!UF3) exceeds grid limits. Max rows: 966, max columns: 27.
But the code get_notes() runs well before this PR, even on a very big sheet.
I think the modifications about get_notes when grid_range=None reduces the limitation significantly.
So in my opinion, When grid_range is not given, using the previous code is a better way.

muddi900 · 2024-07-01T12:54:26Z

Do you have a public big sheet I can test this on?

nbwzx · 2024-07-01T13:05:14Z

Do you have a public big sheet I can test this on?

The one I mentioned is

https://docs.google.com/spreadsheets/d/158F-jyu8ld8kbdD4I_Lqy4fsX5vvp_MQOq7Ofg8NdSI/

on Sheet name UF3.

muddi900 · 2024-07-03T12:35:48Z

I am getting a different error when using the latest error.

APIError                                  Traceback (most recent call last)
Cell In[10], [line 1](vscode-notebook-cell:?execution_count=10&line=1)
----> [1](vscode-notebook-cell:?execution_count=10&line=1) notes = ws.get_notes()

File ~/src/gspread/gspread/worksheet.py:2665, in Worksheet.get_notes(self, default_empty_value, grid_range)
   [2614](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2614) """Returns a list of lists containing all notes in the sheet or range.
   [2615](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2615) 
   [2616](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2616) .. note::
   (...)
   [2654](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2654)     ]
   [2655](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2655) """
   [2656](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2656) params: ParamsType = {
   [2657](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2657)     "fields": "sheets.data.rowData.values.note",
   [2658](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2658)     "ranges": (
   (...)
   [2662](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2662)     ),
   [2663](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2663) }
-> [2665](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2665) res = self.client.spreadsheets_get(self.spreadsheet_id, params)
   [2667](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2667) # access 0th sheet because we specified a sheet with params["ranges"] above
   [2668](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/worksheet.py:2668) data = res["sheets"][0]["data"][0].get("rowData", [{}])

File ~/src/gspread/gspread/http_client.py:274, in HTTPClient.spreadsheets_get(self, id, params)
    [272](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:272) """A method stub that directly calls `spreadsheets.get <[https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/get>`_](https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/get%3E%60_)."""
    [273](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:273) url = SPREADSHEET_URL % id
--> [274](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:274) r = self.request("get", url, params=params)
    [275](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:275) return r.json()

File ~/src/gspread/gspread/http_client.py:128, in HTTPClient.request(self, method, endpoint, params, data, json, files, headers)
    [126](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:126)     return response
    [127](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:127) else:
--> [128](https://file+.vscode-resource.vscode-cdn.net/Users/mudassirchapra/src/gspread/~/src/gspread/gspread/http_client.py:128)     raise APIError(response)

APIError: APIError: [400]: Range (Introduction!UF3) exceeds grid limits. Max rows: 966, max columns: 27

Since the method was on the Worksheet instance, there was no way to get the wrong index on self.index. I think the original way would be better.

right now, the request params passthrough "{first_worksheet_name}!{self.title}" as the range, which causes the error.

gspread/worksheet.py

lavigne958 · 2024-07-03T21:57:04Z

Thank you everyone for the hard work and the all the edge case testing ! we managed to catch a very particular use case that we would not see otherwise 💪

I tried it and it works fine, it is well explained that the resulting matrix is not square.
it's quite complicated for a novice to use the example with fill_gaps I believe, should we simply offer an option to fill the gaps for the end user ? 🤔

something like:

option name: auto_fill_gaps
option type: bool
option default value: False (in order to prevent breaking change_

that requires us to do some computation based on the resulting matrix, meaning on a big big sheet we need to iterate over all rows to find the longest row before iterating again on all rows to expand them.

I'm fine with it, what do you guys think about it ?

nbwzx · 2024-07-03T23:55:58Z

I agree. I think it is not convenient for users to copy the relevant code from the documentation.

muddi900 · 2024-07-04T11:14:41Z

@alifeee Sorry for the git reset shenanigans. I missed @lavigne958 comment, and tried to redo everything.

I have incorporated the changes suggested in the comment.

@lavigne958

…t names. (via @lavigne958)

alifeee · 2024-07-05T14:19:04Z

@alifeee Sorry for the git reset shenanigans. I missed @lavigne958 comment, and tried to redo everything.

no worries :) I have refreshed the test cassette (by deleting tests/cassettes/WorksheetTest.test_get_notes.json and re-running the test with GS_RECORD_MODE=all)

Also, the git reset has removed the extra test that I made for 2nd sheet notes. Please could you add this back? You can see it in here:

muddi900/gspread@add/notes-range...burnash:gspread:add/notes-range-tests

lavigne958

the changes are good.
The CI is failing due to outdated/missing cassette.
Please remove the recorded cassette: tests/cassettes/WorksheetTest.test_get_notes.json and record the cassettes again.

You can use the value new_episodes to only record the missing cassette.

.github/CONTRIBUTING.md

lavigne958

your tests are still failing, please remove the file tests/cassettes/WorksheetTest.test_get_notes.json and run the test with the following command: GS_RECORD_MODE='new_episodes' CREDS_FILENAME=<path to credentials> tox -e py

alifeee · 2024-07-07T09:55:31Z

the test test_get_notes_2nd_sheet is still missing, please see #1487 (comment) for context :)

tests/worksheet_test.py

muddi900 · 2024-07-16T08:33:16Z

So should autofill be a different issue. Because that would be a rewrite of the method.

lavigne958 · 2024-07-17T21:22:09Z

So should autofill be a different issue. Because that would be a rewrite of the method.

I am sorry I don't understand your comment, could you please elaborate ?

muddi900 · 2024-07-19T09:32:21Z

Thank you everyone for the hard work and the all the edge case testing ! we managed to catch a very particular use case that we would not see otherwise 💪

I tried it and it works fine, it is well explained that the resulting matrix is not square. it's quite complicated for a novice to use the example with fill_gaps I believe, should we simply offer an option to fill the gaps for the end user ? 🤔

something like:
* option name: `auto_fill_gaps`

* option type: bool

* option default value: `False` (in order to prevent breaking change_
that requires us to do some computation based on the resulting matrix, meaning on a big big sheet we need to iterate over all rows to find the longest row before iterating again on all rows to expand them.

I'm fine with it, what do you guys think about it ?

I was referring to this

lavigne958 · 2024-07-23T20:05:25Z

I see, thank you.
we can make it 2 different issues, though we can't make a release between the time each issue is merged, we want to provide the feature once and not provide the feature then provide a different way to use it.

lavigne958 · 2024-07-31T17:02:04Z

I come back to the point about fill_gaps, the current feature requires the user to call fill_gaps. so for now we can't change the the output, we can merge this as is, once the cassettes have been updated and the no-op line has been removed.

If you don't have time or struggle with the cassettes @muddi900, just let me know and I'll cherry-pick you commits, you'll be mentioned as contributor still and we'll merge these changes soon.

muddi900 and others added 3 commits June 23, 2024 19:10

Added range option for get_notes.[Issue burnash#1482]

2c6270c

updated docstring

4d4e592

updated test_get_notes to handle range.

1368fba

alifeee changed the title ~~Added a range option to Worksheet.get_range [Issue #1482]~~ Added a range option to Worksheet.get_notes [Issue #1482] Jun 29, 2024

alifeee requested a review from lavigne958 June 29, 2024 15:16

alifeee assigned alifeee and muddi900 Jun 29, 2024

alifeee added the Feature Request label Jun 29, 2024

alifeee added this to the 6.2.0 milestone Jun 29, 2024

lavigne958 requested changes Jul 3, 2024

View reviewed changes

gspread/worksheet.py Outdated Show resolved Hide resolved

muddi900 force-pushed the add/notes-range branch from cdc0f83 to 1368fba Compare July 4, 2024 10:44

muddi900 added 2 commits July 4, 2024 11:16

reverted all changes made by @alifeee

5951c46

passing absolute_range_name to ranges to handle cell-name like shee…

c0f5a68

…t names. (via @lavigne958)

muddi900 force-pushed the add/notes-range branch from 21ce6fa to c0f5a68 Compare July 4, 2024 11:16

alifeee added 2 commits July 5, 2024 15:14

refresh test cassette

7303c8f

refresh test cassette (again...)

1816e9a

Added range notes test

cd2a1eb

lavigne958 reviewed Jul 5, 2024

View reviewed changes

.github/CONTRIBUTING.md Outdated Show resolved Hide resolved

.github/CONTRIBUTING.md Outdated Show resolved Hide resolved

reverting CONTRIBUTING.md

6e395aa

lavigne958 reviewed Jul 6, 2024

View reviewed changes

muddi900 added 2 commits July 8, 2024 10:17

added test_get_notes_2nd_sheet (authored by @alifeee)

0b44118

removed the cassettes

1a3e5f4

lavigne958 reviewed Jul 8, 2024

View reviewed changes

tests/worksheet_test.py Outdated Show resolved Hide resolved

removed the no-op line

f31db23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a range option to `Worksheet.get_notes` [Issue #1482] #1487

Added a range option to `Worksheet.get_notes` [Issue #1482] #1487

muddi900 commented Jun 24, 2024

alifeee commented Jun 25, 2024

nbwzx commented Jun 29, 2024

alifeee commented Jun 29, 2024

alifeee commented Jun 29, 2024

muddi900 commented Jun 30, 2024

nbwzx commented Jul 1, 2024

muddi900 commented Jul 1, 2024

nbwzx commented Jul 1, 2024

muddi900 commented Jul 3, 2024

lavigne958 commented Jul 3, 2024

nbwzx commented Jul 3, 2024

muddi900 commented Jul 4, 2024

alifeee commented Jul 5, 2024

lavigne958 left a comment

lavigne958 left a comment

alifeee commented Jul 7, 2024

muddi900 commented Jul 16, 2024

lavigne958 commented Jul 17, 2024

muddi900 commented Jul 19, 2024

lavigne958 commented Jul 23, 2024

lavigne958 commented Jul 31, 2024

Added a range option to Worksheet.get_notes [Issue #1482] #1487

Are you sure you want to change the base?

Added a range option to Worksheet.get_notes [Issue #1482] #1487

Conversation

muddi900 commented Jun 24, 2024

alifeee commented Jun 25, 2024

nbwzx commented Jun 29, 2024

alifeee commented Jun 29, 2024

alifeee commented Jun 29, 2024

muddi900 commented Jun 30, 2024

nbwzx commented Jul 1, 2024

muddi900 commented Jul 1, 2024

nbwzx commented Jul 1, 2024

muddi900 commented Jul 3, 2024

lavigne958 commented Jul 3, 2024

nbwzx commented Jul 3, 2024

muddi900 commented Jul 4, 2024

alifeee commented Jul 5, 2024

lavigne958 left a comment

Choose a reason for hiding this comment

lavigne958 left a comment

Choose a reason for hiding this comment

alifeee commented Jul 7, 2024

muddi900 commented Jul 16, 2024

lavigne958 commented Jul 17, 2024

muddi900 commented Jul 19, 2024

lavigne958 commented Jul 23, 2024

lavigne958 commented Jul 31, 2024

Added a range option to `Worksheet.get_notes` [Issue #1482] #1487

Added a range option to `Worksheet.get_notes` [Issue #1482] #1487