pandas.merge#
pandas. merge ( left , right , how = ‘inner’ , on = None , left_on = None , right_on = None , left_index = False , right_index = False , sort = False , suffixes = (‘_x’, ‘_y’) , copy = None , indicator = False , validate = None ) [source] #
Merge DataFrame or named Series objects with a database-style join.
A named Series object is treated as a DataFrame with a single named column.
The join is done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. When performing a cross merge, no column specifications to merge on are allowed.
If both key columns contain rows where the key is a null value, those rows will be matched against each other. This is different from usual SQL join behaviour and can lead to unexpected results.
Parameters left DataFrame or named Series right DataFrame or named Series
Type of merge to be performed.
- left: use only keys from left frame, similar to a SQL left outer join; preserve key order.
- right: use only keys from right frame, similar to a SQL right outer join; preserve key order.
- outer: use union of keys from both frames, similar to a SQL full outer join; sort keys lexicographically.
- inner: use intersection of keys from both frames, similar to a SQL inner join; preserve the order of the left keys.
- cross: creates the cartesian product from both frames, preserves the order of the left keys.
Column or index level names to join on. These must be found in both DataFrames. If on is None and not merging on indexes then this defaults to the intersection of the columns in both DataFrames.
left_on label or list, or array-like
Column or index level names to join on in the left DataFrame. Can also be an array or list of arrays of the length of the left DataFrame. These arrays are treated as if they are columns.
right_on label or list, or array-like
Column or index level names to join on in the right DataFrame. Can also be an array or list of arrays of the length of the right DataFrame. These arrays are treated as if they are columns.
left_index bool, default False
Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels.
right_index bool, default False
Use the index from the right DataFrame as the join key. Same caveats as left_index.
sort bool, default False
Sort the join keys lexicographically in the result DataFrame. If False, the order of the join keys depends on the join type (how keyword).
suffixes list-like, default is (“_x”, “_y”)
A length-2 sequence where each element is optionally a string indicating the suffix to add to overlapping column names in left and right respectively. Pass a value of None instead of a string to indicate that the column name from left or right should be left as-is, with no suffix. At least one of the values must not be None.
copy bool, default True
If False, avoid copy if possible.
indicator bool or str, default False
If True, adds a column to the output DataFrame called “_merge” with information on the source of each row. The column can be given a different name by providing a string argument. The column will have a Categorical type with the value of “left_only” for observations whose merge key only appears in the left DataFrame, “right_only” for observations whose merge key only appears in the right DataFrame, and “both” if the observation’s merge key is found in both DataFrames.
validate str, optional
If specified, checks if merge is of specified type.
- “one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.
- “one_to_many” or “1:m”: check if merge keys are unique in left dataset.
- “many_to_one” or “m:1”: check if merge keys are unique in right dataset.
- “many_to_many” or “m:m”: allowed, but does not result in checks.
A DataFrame of the two merged objects.
Merge with optional filling/interpolation.
Similar method using indices.
Support for specifying index levels as the on , left_on , and right_on parameters was added in version 0.23.0 Support for merging named Series objects was added in version 0.24.0
>>> df1 = pd.DataFrame('lkey': ['foo', 'bar', 'baz', 'foo'], . 'value': [1, 2, 3, 5]>) >>> df2 = pd.DataFrame('rkey': ['foo', 'bar', 'baz', 'foo'], . 'value': [5, 6, 7, 8]>) >>> df1 lkey value 0 foo 1 1 bar 2 2 baz 3 3 foo 5 >>> df2 rkey value 0 foo 5 1 bar 6 2 baz 7 3 foo 8
Merge df1 and df2 on the lkey and rkey columns. The value columns have the default suffixes, _x and _y, appended.
>>> df1.merge(df2, left_on='lkey', right_on='rkey') lkey value_x rkey value_y 0 foo 1 foo 5 1 foo 1 foo 8 2 foo 5 foo 5 3 foo 5 foo 8 4 bar 2 bar 6 5 baz 3 baz 7
Merge DataFrames df1 and df2 with specified left and right suffixes appended to any overlapping columns.
>>> df1.merge(df2, left_on='lkey', right_on='rkey', . suffixes=('_left', '_right')) lkey value_left rkey value_right 0 foo 1 foo 5 1 foo 1 foo 8 2 foo 5 foo 5 3 foo 5 foo 8 4 bar 2 bar 6 5 baz 3 baz 7
Merge DataFrames df1 and df2, but raise an exception if the DataFrames have any overlapping columns.
>>> df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False)) Traceback (most recent call last): . ValueError: columns overlap but no suffix specified: Index(['value'], dtype='object')
>>> df1 = pd.DataFrame('a': ['foo', 'bar'], 'b': [1, 2]>) >>> df2 = pd.DataFrame('a': ['foo', 'baz'], 'c': [3, 4]>) >>> df1 a b 0 foo 1 1 bar 2 >>> df2 a c 0 foo 3 1 baz 4
>>> df1.merge(df2, how='inner', on='a') a b c 0 foo 1 3
>>> df1.merge(df2, how='left', on='a') a b c 0 foo 1 3.0 1 bar 2 NaN
>>> df1 = pd.DataFrame('left': ['foo', 'bar']>) >>> df2 = pd.DataFrame('right': [7, 8]>) >>> df1 left 0 foo 1 bar >>> df2 right 0 7 1 8
>>> df1.merge(df2, how='cross') left right 0 foo 7 1 foo 8 2 bar 7 3 bar 8
pandas.concat#
Can also add a layer of hierarchical indexing on the concatenation axis, which may be useful if the labels are the same (or overlapping) on the passed axis number.
Parameters objs a sequence or mapping of Series or DataFrame objects
If a mapping is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all None in which case a ValueError will be raised.
axis , default 0
The axis to concatenate along.
join , default ‘outer’
How to handle indexes on other axis (or axes).
ignore_index bool, default False
If True, do not use the index values along the concatenation axis. The resulting axis will be labeled 0, …, n — 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Note the index values on the other axes are still respected in the join.
keys sequence, default None
If multiple levels passed, should contain tuples. Construct hierarchical index using the passed keys as the outermost level.
levels list of sequences, default None
Specific levels (unique values) to use for constructing a MultiIndex. Otherwise they will be inferred from the keys.
names list, default None
Names for the levels in the resulting hierarchical index.
verify_integrity bool, default False
Check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation.
sort bool, default False
Sort non-concatenation axis if it is not already aligned.
copy bool, default True
If False, do not copy data unnecessarily.
Returns object, type of objs
When concatenating all Series along the index (axis=0), a Series is returned. When objs contains at least one DataFrame , a DataFrame is returned. When concatenating along the columns (axis=1), a DataFrame is returned.
Join DataFrames using indexes.
Merge DataFrames by indexes or columns.
The keys, levels, and names arguments are all optional.
A walkthrough of how this method fits in with other tools for combining pandas objects can be found here.
It is not recommended to build DataFrames by adding single rows in a for loop. Build a list of rows and make a DataFrame in a single concat.
>>> s1 = pd.Series(['a', 'b']) >>> s2 = pd.Series(['c', 'd']) >>> pd.concat([s1, s2]) 0 a 1 b 0 c 1 d dtype: object
Clear the existing index and reset it in the result by setting the ignore_index option to True .
>>> pd.concat([s1, s2], ignore_index=True) 0 a 1 b 2 c 3 d dtype: object
Add a hierarchical index at the outermost level of the data with the keys option.
>>> pd.concat([s1, s2], keys=['s1', 's2']) s1 0 a 1 b s2 0 c 1 d dtype: object
Label the index keys you create with the names option.
>>> pd.concat([s1, s2], keys=['s1', 's2'], . names=['Series name', 'Row ID']) Series name Row ID s1 0 a 1 b s2 0 c 1 d dtype: object
Combine two DataFrame objects with identical columns.
>>> df1 = pd.DataFrame([['a', 1], ['b', 2]], . columns=['letter', 'number']) >>> df1 letter number 0 a 1 1 b 2 >>> df2 = pd.DataFrame([['c', 3], ['d', 4]], . columns=['letter', 'number']) >>> df2 letter number 0 c 3 1 d 4 >>> pd.concat([df1, df2]) letter number 0 a 1 1 b 2 0 c 3 1 d 4
Combine DataFrame objects with overlapping columns and return everything. Columns outside the intersection will be filled with NaN values.
>>> df3 = pd.DataFrame([['c', 3, 'cat'], ['d', 4, 'dog']], . columns=['letter', 'number', 'animal']) >>> df3 letter number animal 0 c 3 cat 1 d 4 dog >>> pd.concat([df1, df3], sort=False) letter number animal 0 a 1 NaN 1 b 2 NaN 0 c 3 cat 1 d 4 dog
Combine DataFrame objects with overlapping columns and return only those that are shared by passing inner to the join keyword argument.
>>> pd.concat([df1, df3], join="inner") letter number 0 a 1 1 b 2 0 c 3 1 d 4
Combine DataFrame objects horizontally along the x axis by passing in axis=1 .
>>> df4 = pd.DataFrame([['bird', 'polly'], ['monkey', 'george']], . columns=['animal', 'name']) >>> pd.concat([df1, df4], axis=1) letter number animal name 0 a 1 bird polly 1 b 2 monkey george
Prevent the result from including duplicate index values with the verify_integrity option.
>>> df5 = pd.DataFrame([1], index=['a']) >>> df5 0 a 1 >>> df6 = pd.DataFrame([2], index=['a']) >>> df6 0 a 2 >>> pd.concat([df5, df6], verify_integrity=True) Traceback (most recent call last): . ValueError: Indexes have overlapping values: ['a']
Append a single row to the end of a DataFrame object.
>>> df7 = pd.DataFrame('a': 1, 'b': 2>, index=[0]) >>> df7 a b 0 1 2 >>> new_row = pd.Series('a': 3, 'b': 4>) >>> new_row a 3 b 4 dtype: int64 >>> pd.concat([df7, new_row.to_frame().T], ignore_index=True) a b 0 1 2 1 3 4