Query String Parameters
What are query strings? How do I use them? In this lesson you’ll learn exactly that. The .get() function accepts a params argument, which can be a dictionary, a list of tuples or raw bytes. The following example uses the GitHub Search API looking for the requests library:
>>> requests.get( . 'https://api.github.com/search/repositories', . params='q': 'requests+language:python'> . ) Response [200]> agerbes on March 14, 2020
thank you very much for the great tutorial.
I have a special question.
Using the request library I print out the result of the url = ‘www.dropbox.com/h?preview=ophelp.txt‘
I would like to store the content of this url into it’s name = ‘ophelp’.
How can I use the request library to get the name searching for ‘preview=’ and getting ‘ophelp’?
I thought about using the query string parameters as you shown in your tutorial. But I am not sure how to search for it exactly.
Chris Bailey RP Team on March 15, 2020
Hi @agerbes Anja, I’m not sure I completely understand your question. It looks like you want to search for a file on dropbox, and get a preview of the text. I believe searches would be limited to a particular users files. I haven’t used the dropbox api, but I looked it over a bit and you would need to use a different url to use requests with it. The url would be ‘https://api.dropbox.com/’ . I did find some of the resources available from their site. I hope they can help you with what you are trying to accomplish. This link has the http documentation for using the dropbox api, including the available end points: Dropbox for HTTP
This link is to their Python package for doing development. It looks pretty well documented also: Dropbox for Python Developers. I hope some of this helps.
The speed of these videos is quite fast, IMHO. Speed and depth of explanations not really beginner-friendly.
Dan Bader RP Team on April 9, 2020
Thanks for your feedback there @jc123k. This course is aimed at an intermediate skill level, so it isn’t intended to be a beginner’s tutorial. But thank you for the feedback, maybe there’s room for doing a beginner-focused course or tutorial on this topic in the future 🙂
The response I got does not relate to the webpage. Where does the data come from?
Repository name: grequests Repository description: Requests + Gevent =
Sorry to correct what I said this is the response from the repo:
Got it. Did some research on string query and worked out what to use in url: api.github.com/search/repositories?q=schedule+user:dbader
mikesult on May 4, 2020
DaveyD, I also had the same response from search.py
Repository name: grequests Repository description: Requests + Gevent =
if I omitted the ‘+language:python’ part of the query then ‘requests’ is positioned at index 0 of ‘items’. (params=). it seems true for today anyway.
Repository name: requests Repository description: A simple, yet elegant HTTP library.
I’m guessing that just means the repository structure at github has changed a little since the video was made. Since we are looking for index 0 of the items both results are successful queries. Just not what we expected.
mikesult on May 4, 2020
Thinking more about why grequests came out at index 0. some questions:
- When using multiple params are they ‘or’ed together or ‘and’ed together? I was thinkng that you added ‘+language:python’ as an ‘and’ type of filter in case there were other repos named ‘requests’ from another language and you didn’t want them in the response.
- Since grequests was a match I infer that query ‘requests’ will match any name that contains ‘requests’? i.e. fiddlerequests, frequests lorequests. Or maybe that match was because it is a python language repo? I’m confused as to why grequests matched?
- Is json.items() an alphabetically ordering?, i.e. if there was a repo name arequests would that bump grequests down.?
Chris Bailey RP Team on May 5, 2020
Hi @DaveyD and @mikesult, Wow. A lot has changed in the last year, not only for the results of this search but for the “requests” package. It is now under the Python Software Foundation the PSF.
You are both right that existing search now returns ‘grequests’ as the first item. And you are correct, if you remove the ‘+language:python’ part it will now return ‘requests’ as a first item. Not sure why it is not under ‘Python’ for the language, weird.
I was thinking if you want to see more of the search results you could add a simple for loop.
# at the end of the file change it to json_response = response.json() for i in range(len(json_response['items')): repository = json_response['items'][i] print(f'i>: Respository name: repository["name"]>') print(f'i>: Respository description: repository["description"]>') This would give you list of the returned search items.
To answer your other questions
1: The ‘+’ works like an ‘AND’ statement, so it will reduce your results. In this case narrowing to only the keyword language, with Python. You can try this search out directly from the search bar on the actual site github.com itself. Another keywords as I show later is ‘user:’ and I just tried out another ‘topic:’
2: You are right that it will look within the name of the repository but it also looks at (searches) the description text for the word. It’s truly doing a search.
3: It is a ranking, not alpha. So it depends on how github orders the results.
I hope this helps. Any site will have their own scheme for how this will work. And it will be a moving target for a video to capture results of a search for the moment in time.
Tonya Sims on July 23, 2020
Hi Chris and Real Python community! Your tutorial is amazing and I’m learning a lot. Quick question about the query string for the first example. When I go to this url in my browser: api.github.com/search/respositories?q=requests+language:python
I read through the comments here and just want to verify that it won’t work anymore because Github moved it to another repository? Is that correct or am I just misunderstanding?
Tonya Sims on July 23, 2020
Hi again, about my previous comment. When I run the code for:
I’m getting JSON data back, so maybe the way I constructed the URL is incorrect: api.github.com/search/respositories?q=requests+language:python
Chris Bailey RP Team on July 23, 2020
As you noticed in my earlier comments, the URL scheme seems to be changed a bit for where to find “requests”. Here is a URL scheme that seems to work now. https://api.github.com/search/repositories?q=requests+org:psf . It looks like the “full name” is now psf/requests. The types of things you can specify in the query string seem to really vary over time. “org” for organizations, language, etc. You can also get an idea if you want to see the types of URL formats, from this URL: https://api.github.com , it will return JSON of the end points to explore. And the documentation is also here.
mikelisfbay on July 30, 2020
I found the root cause of the problem. It is due to percent encoding scheme performed by the requests package on “+” and “:”.
The solution is to pass the string directly.
1. Original solution with the dictionary fromatted payload
Output: (No good) api.github.com/search/repositories?q=requests%2Blanguage%3Apython&sort=stars&order=desc Repository name: httpbin Repository description: HTTP Request & Response Service, written in Python + Flask.
2. New solution with the direct string formatted payload
Output: (This is what we want to see) api.github.com/search/repositories?q=requests+language:python&sort=stars&order=desc Repository name: requests Repository description: A simple, yet elegant HTTP library.
3. How to convert a dictionary to a string. (the practical solution)
payload_str = “&”.join(“%s=%s” % (k,v) for k,v in payload.items())
r = requests.get(url, params=payload_str) print(r.url)
The above solution is contributed by a user named furas on Stackoverflow.
jramirez857 on Dec. 30, 2021
Thanks for the solution @mikelisfbay! I have used your proposed solution in the code below to get the requests repository:
json_response = response.json() repository = json_response[‘items’][0]
Python requests: GET Request Explained
In this tutorial, you’ll learn how to use the Python requests library’s get method to fetch data via HTTP. The Python requests library abstracts the complexities in making HTTP requests. The requests.get() method allows you to fetch an HTTP response and analyze it in different ways.
By the end of this tutorial, you’ll have learned:
- How the Python requests get method works
- How to customize the Python requests get method with headers
- How to use the Python response objects
Understanding the Python requests get Function
An HTTP GET request is used to retrieve data from the specified resource, such as a website. When using the Python requests library, you can use the .get() function to create a GET request for a specified resource.
The function accepts a number of different parameters. Let’s take a look at the function and the different parameters that it accepts:
# Understand the Python requests.get() Function import requests req = requests.get( url, params=None, **kwargs )We can see in the code block above, that the function accepts two parameters:
- A url , which points to a resource, and
- params , which accepts a dictionary or tuples to send in the query string
The .get() function is actually a convenience function based off of the .request() function. The optional keyword arguments that you can pass in derive from that function. Let’s take a look at the options that you can pass in: