Python requests cookie in file

How to save requests (python) cookies to a file?

to keep all the cookies in a file and then restore the cookies from a file.

There is no immediate way to do so, but it’s not hard to do.

You can get a CookieJar object from the session with session.cookies , and use pickle to store it to a file.

import requests, pickle session = requests.session() # Make some calls with open('somefile', 'wb') as f: pickle.dump(session.cookies, f) 

session = requests.session() # or an existing session with open('somefile', 'rb') as f: session.cookies.update(pickle.load(f)) 

The requests library uses the requests.cookies.RequestsCookieJar() subclass, which explicitly supports pickling and a dict-like API. The RequestsCookieJar.update() method can be used to update an existing session cookie jar with the cookies loaded from the pickle file.

After a call such as r = requests.get() , r.cookies will return a RequestsCookieJar which you can directly pickle , i.e.

import pickle def save_cookies(requests_cookiejar, filename): with open(filename, 'wb') as f: pickle.dump(requests_cookiejar, f) def load_cookies(filename): with open(filename, 'rb') as f: return pickle.load(f) #save cookies r = requests.get(url) save_cookies(r.cookies, filename) #load cookies and do a request requests.get(url, cookies=load_cookies(filename)) 

If you want to save your cookies in human-readable format, you have to do some work to extract the RequestsCookieJar to a LWPCookieJar .

import cookielib def save_cookies_lwp(cookiejar, filename): lwp_cookiejar = cookielib.LWPCookieJar() for c in cookiejar: args = dict(vars(c).items()) args['rest'] = args['_rest'] del args['_rest'] c = cookielib.Cookie(**args) lwp_cookiejar.set_cookie(c) lwp_cookiejar.save(filename, ignore_discard=True) def load_cookies_from_lwp(filename): lwp_cookiejar = cookielib.LWPCookieJar() lwp_cookiejar.load(filename, ignore_discard=True) return lwp_cookiejar #save human-readable r = requests.get(url) save_cookies_lwp(r.cookies, filename) #you can pass a LWPCookieJar directly to requests requests.get(url, cookies=load_cookies_from_lwp(filename)) 

Expanding on @miracle2k’s answer, requests Session s are documented to work with any cookielib CookieJar . The LWPCookieJar (and MozillaCookieJar ) can save and load their cookies to and from a file. Here is a complete code snippet which will save and load cookies for a requests session. The ignore_discard parameter is used to work with httpbin for the test, but you may not want to include it your in real code.

import os from cookielib import LWPCookieJar import requests s = requests.Session() s.cookies = LWPCookieJar('cookiejar') if not os.path.exists('cookiejar'): # Create a new cookies file and set our Session's cookies print('setting cookies') s.cookies.save() r = s.get('http://httpbin.org/cookies/set?k1=v1&k2=v2') else: # Load saved cookies from the file and use them in a request print('loading saved cookies') s.cookies.load(ignore_discard=True) r = s.get('http://httpbin.org/cookies') print(r.text) # Save the session's cookies back to the file s.cookies.save(ignore_discard=True) 

import json with open('cookie.txt', 'w') as f: json.dump(requests.utils.dict_from_cookiejar(bot.cookies), f) 

import json session = requests.session() # or an existing session with open('cookie.txt', 'r') as f: cookies = requests.utils.cookiejar_from_dict(json.load(f)) session.cookies.update(cookies) 

session.cookies = LWPCookieJar('cookies.txt') 

The CookieJar API requires you to call load() and save() manually though. If you do not care about the cookies.txt format, I have a ShelvedCookieJar implementation that will persist on change.

I found that the other answers had problems:

• They didn’t apply to sessions.
• They didn’t save and load properly. Only the cookie name and value was saved, the expiry date, domain name, etc. was all lost.

This answer fixes these two issues:

import requests.cookies def save_cookies(session, filename): if not os.path.isdir(os.path.dirname(filename)): return False with open(filename, 'w') as f: f.truncate() pickle.dump(session.cookies._cookies, f) def load_cookies(session, filename): if not os.path.isfile(filename): return False with open(filename) as f: cookies = pickle.load(f) if cookies: jar = requests.cookies.RequestsCookieJar() jar._cookies = cookies session.cookies = jar else: return False 

Then just call save_cookies(session, filename) to save or load_cookies(session, filename) to load. Simple as that.

You can pickle the cookies object directly:

cookies = pickle.dumps(session.cookies) 

The dict representation misses a lot of informations: expiration, domain, path…

It depends on the usage you intend to do with the cookies, but if you don’t have informations about the expiration, for example, you should implement the logic to track expiration by hand.

Pickling the library returned object lets you easily reconstruct the state, then you can relay on the library implementation.

Obviously, this way, the consumer of the pickled object needs to use the same library

Simple way to convert cookies into list of dicts and save to json or db.
This is methods of class which have session attribute.

def dump_cookies(self): cookies = [] for c in self.session.cookies: cookies.append(< "name": c.name, "value": c.value, "domain": c.domain, "path": c.path, "expires": c.expires >) return cookies def load_cookies(self, cookies): for c in cookies: self.session.cookies.set(**c) 

All we need is five parameters such as: name , value , domain , path , expires

Источник

Python requests cookie in file

Last updated: Jul 2, 2023
Reading time · 4 min

# Table of Contents

# Python: How to get and set Cookies when using Requests

Use the Session class to set and get cookies when using the requests module in Python.

The class creates a Session object that stores the cookies and all requests that are made handle cookies automatically.

Copied! 
import requests session = requests.Session() print(session.cookies.get_dict()) # <> response = session.get('http://google.com') # print(session.cookies.get_dict()) 

Make sure you have the requests module installed to be able to run the code sample.

Copied! 
pip install requests # or with pip3 pip3 install requests 

The Session object enables you to persist cookies across requests.

The object persists cookies across all requests that were made using the Session instance.

The Session object has all of the methods of the main requests API.

The get_dict() method returns a Python dictionary of the name-value pairs of cookies.

You can also use Sessions as context managers.

Copied! 
import requests with requests.Session() as session: print(session.cookies.get_dict()) # <> response = session.get('http://google.com') # print(session.cookies.get_dict()) 

Make sure the code that accesses the Session object is inside the indented block.

# Accessing the path and the domain when using Cookies with requests

It is very likely, that you will also have to access the domain and the path to which the request was made when accessing the cookies.

You can use a list comprehension to construct a list of dictionaries that contain the path and domain.

Copied! 
import requests session = requests.Session() print(session.cookies.get_dict()) # <> print('-' * 50) response = session.get('http://google.com') # print(session.cookies.get_dict()) print('-' * 50) result = [ 'name': c.name, 'value': c.value, 'domain': c.domain, 'path': c.path> for c in session.cookies ] # [] print(result) 

We used a list comprehension to iterate over the RequestCookieJar object ( session.cookies ) and returned a dictionary on each iteration.

The dictionary contains the name and value of the cookie, the path and the domain.

# Sending cookies with a request

If you want to send cookies with a request, set the cookies keyword argument.

Copied! 
import requests session = requests.Session() response = session.get( 'https://httpbin.org/cookies', cookies='my-cookie': 'my-value'> ) # # "cookies": # "my-cookie": "my-value" # > # > print(response.text) 

We set the cookies keyword argument to a dictionary of key-value pairs.

# Accessing the cookies attribute directly on the Response object

In more recent versions of the requests module, you can also access the cookies attribute directly on the Response object.

Copied! 
import requests response = requests.get('http://google.com', timeout=30) # print(response.cookies.get_dict()) result = [ 'name': c.name, 'value': c.value, 'domain': c.domain, 'path': c.path> for c in response.cookies ] # [] print(result) 

Notice that we didn’t instantiate the Session class.

We simply accessed the cookies attribute on the Response object and called the get_dict() method on the RequestCookieJar object.

However, the management of cookies is automated when you use a Session object.

This means that you won’t have to send the cookies explicitly:

Copied! 
import requests session = requests.Session() response = session.get( 'https://httpbin.org/cookies', cookies='my-cookie': 'my-value'> ) # # "cookies": # "my-cookie": "my-value" # > # > print(response.text) 

Because it will be done for your automatically.

# Saving the requests cookies in a file and restoring them

You can also save the requests cookies in a file.

Copied! 
import json import requests response = requests.get('http://google.com', timeout=30) with open('cookies.txt', 'w', encoding='utf-8') as f: json.dump( requests.utils.dict_from_cookiejar(response.cookies), f ) 

The cookies.txt file stores the following JSON string.

Copied! 
"AEC": "Ad49MVEu4N64Tk1gEROw417s9FgcqdqeIeVZ8eL9m-HQldzOrLAF2HvxHQ"> 

Notice that we used the requests.utils.dict_from_cookiejar method to create a dictionary from the RequestCookieJar object.

We then passed the dictionary and the file object to the json.dump method.

The json.dump method serializes the supplied object as a JSON formatted stream and writes it to a file.

You can then read and restore the cookies from the file.

Copied! 
import json import requests session = requests.session() with open('cookies.txt', 'r', encoding='utf-8') as f: cookies = requests.utils.cookiejar_from_dict(json.load(f)) session.cookies.update(cookies) # print(session.cookies.get_dict()) 

We created a brand new Session object but you can use an existing Session .

1. We used the json.load() method to convert the contents of the file to a native Python dictionary.
2. We used the requests.utils.cookiejar_from_dict method to create a RequestCookieJar object from the dictionary.
3. The last step is to update the cookies of the Session object with the newly read cookies.

# Additional Resources

You can learn more about the related topics by checking out the following tutorials:

I wrote a book in which I share everything I know about how to become a better, more efficient programmer.

Источник

Как в requests получить куки, правильно сохранить в файл, потом использовать?

Здравствуйте, периодически мне нужно будет запускать скрипт, который какое-то время будет парсить страницы НЕ моего сайта. Так как глубже авторизации на сайт меня не пустит, мне нужно залогиниться на сайте, и идти вглубь. Для этого я использую Requests Session. Мой скрипт по парсингу-то готов, однако, если я буду периодически заходить и логиниться, и снова логиниться, админы-то не дураки, спалят. Тут я подумал о безопасности, получается мне нужно каким-то образом заходить и не палиться тем, что логинюсь, а для этого и существуют куки. Но с ними проблема. Если после логина на сайте использовать команду

cookie1 = requests.utils.dict_from_cookiejar(session.cookies)

Ну ладно. Получили, допустим сохранили (К этому еще надо вернуться) в файл, запустил скрипт, скрипт нашел файл с куками (не важно какими) с помощью команды

try: session.cookies.set_cookie(cookie1) except: session.cookies.set_cookie(cookie2)

пытаюсь вставить куки, но в первом случае выдает
AttributeError(«‘dict’ object has no attribute ‘value'»,)
Во втором
AttributeError(«‘RequestsCookieJar’ object has no attribute ‘value'»,)
Что не нравится requests? Сам же дал эти куки, как их блин использовать?
Еще вопрос как сохранять куки, пробовал в файлы json, но толку ноль, он потом их прочитать нормально не может

Средний 2 комментария

Источник