Retrieve more than 1000 records using Python

API & Customization
1

Hi, I wrote a python programm to retrieve and update records directly through the API. The output is 1000 records max but my data object contains more records. In the Knack-knowledge database it reads:
Note that at this time, to retrieve more than 1000 records, you will need to programmatically loop through your object utilizing our Pagination, until all required records are retrieved.

Since I am a newbie at both - Knack and Python - I would appreciate if someone could help me how to do that!

Thank you!!

2

Can’t help you on this one I’m afraid, will need a coder to jump in :blush:

3

Hey Sofie. I have a few apps where I work on similar processes as yours except that I mostly work with Javascript. I’m going to leave the code snippet for my nodejs function below for you to see how I achieve this. You should be able to adopt from that.

utils.knackGet = async (object, recordID, noOfRecords, filters, getAllRecords, pageNum) => {
  try {
    if(recordID !== "") {
      var linkEnd = "records/" + recordID
    } else {
      var linkEnd = "records"
    }
    const pages = await axios.get(`https://api.knack.com/v1/objects/` + object + `/` + linkEnd + `?`, {
      params: {
        filters: ((filters)),
        rows_per_page: noOfRecords,
        page: pageNum
      },
      headers: {
        "X-Knack-REST-API-Key": `${process.env.KNACKAPIKEY}`,
        "X-Knack-Application-Id": `${process.env.KNACKAPPID}`
      }
    });
    var noOfpages = pages.data.total_pages;
    if (recordID !== "") {
      records = pages.data;
    } else {
      records = pages.data.records;
    }
    if (getAllRecords === true && noOfpages > 1) {
      for (var i = 2; i <= noOfpages; i++) {
        var pages2 = await axios.get(`https://api.knack.com/v1/objects/` + object + `/records?`, {
          params: {
            filters: ((filters)),
            rows_per_page: noOfRecords,
            page: i
          },
          headers: {
            "X-Knack-REST-API-Key": `${process.env.KNACKAPIKEY}`,
            "X-Knack-Application-Id": `${process.env.KNACKAPPID}`
          }
        });
        for (var j = 0; j < pages2.data.records.length; j++) {
          records.push(pages2.data.records[j]);
        }
      }
    }
    return records;
  } catch (err) {
    throw err;
  }
}

If you’re having trouble with it, can you post your code snippet below and I can help tweak it… I have a little experience working with Python…

1 Like
4

@Sofie,
as you know there is a limit of 1000 records to retrieve per api call, so assuming you are mentioning rows_per_page=1000. and in your initial call you might have page=1 in your query string.

so your api url should be: https://api.knack.com/v1/objects/[your object name]/records?page=1&rows_per_page=1000

now to retrieve remaining or more than thousand records, you need to make an another ajax call with url something like this:
https://api.knack.com/v1/objects/[your object name]/records?page=2&rows_per_page=1000
which will retrieve remaining or next set of records.

Thanks!