Supported Data Spark DataFrame APIs

The C3 Agentic AI Platform supports many Pandas APIs on Spark through Data Spark to perform dataset operations.

Below is a list of supported DataFrame APIs. Functionality is similar to functionality from Pandas. Methods annotated with an * are partially supported.

API Name	Link to Pandas on Spark 3.5.8	Comment
`T`	T
`abs`	abs
`add`	add
`add_prefix`	add_prefix
`add_suffix`	add_suffix
`agg`	agg
`all`	all
`any`	any
`append`	append
`*apply`	apply	Doesn't support external reference inside lambda function
`*applymap`	applymap	Doesn't support external reference inside lambda function
`astype`	astype
`at_time`	at_time
`backfill`	backfill
`between_time`	between_time
`bfill`	bfill
`bool`	bool
`clip`	clip
`columns`	columns
`combine_first`	combine_first
`copy`	copy
`corr`	corr
`corrwith`	corrwith
`count`	count
`cov`	cov
`cummax`	cummax
`cummin`	cummin
`cumprod`	cumprod
`cumsum`	cumsum
`describe`	describe
`diff`	diff
`div`	div
`divide`	divide
`dot`	dot
`drop`	drop
`drop_duplicates`	drop_duplicates
`droplevel`	droplevel
`dropna`	dropna
`duplicated`	duplicated
`empty`	empty
`eq`	eq
`equals`	equals
`eval`	eval
`explode`	explode
`ffill`	ffill
`fillna`	fillna
`filter`	filter
`first`	first
`first_valid_index`	first_valid_index
`floordiv`	floordiv
`ge`	ge
`get`	get
`groupby`	groupby
`gt`	gt
`head`	head
`idxmax`	idxmax
`idxmin`	idxmin
`*iloc`	iloc	Only supports get values. Set values have no effect.
`index`	index
`insert`	insert
`interpolate`	interpolate
`isin`	isin
`isna`	isna
`isnull`	isnull
`items`	items
`iteritems`	iteritems
`iterrows`	iterrows
`itertuples`	itertuples
`join`	join
`keys`	keys
`kurt`	kurt
`kurtosis`	kurtosis
`last`	last
`last_valid_index`	last_valid_index
`le`	le
`loc`	loc	Only supports get values. Set values have no effect.
`lt`	lt
`mad`	mad
`mask`	mask
`max`	max
`mean`	mean
`median`	median
`melt`	melt
`merge`	merge
`min`	min
`mod`	mod
`mode`	mode
`mul`	mul
`ndim`	ndim
`ne`	ne
`nlargest`	nlargest
`notna`	notna
`notnull`	notnull
`nsmallest`	nsmallest
`nunique`	nunique
`pad`	pad
`pct_change`	pct_change
`*pipe`	pipe	Doesn't support external reference inside lambda function
`pivot`	pivot
`pivot_table`	pivot_table
`pop`	pop
`pow`	pow
`prod`	prod
`product`	product
`quantile`	quantile
`query`	query
`radd`	radd
`rank`	rank
`rdiv`	rdiv
`reindex`	reindex
`reindex_like`	reindex_like
`rename`	rename
`rename_axis`	rename_axis
`replace`	replace
`resample`	resample
`reset_index`	reset_index
`rfloordiv`	rfloordiv
`rmod`	rmod
`rmul`	rmul
`round`	round
`rpow`	rpow
`rsub`	rsub
`rtruediv`	rtruediv
`sample`	sample
`select_dtypes`	select_dtypes
`sem`	sem
`set_index`	set_index
`shape`	shape
`shift`	shift
`size`	size
`skew`	skew
`sort_index`	sort_index
`sort_values`	sort_values
`squeeze`	squeeze
`stack`	stack
`std`	std
`sub`	sub
`sum`	sum
`swapaxes`	swapaxes
`swaplevel`	swaplevel
`tail`	tail
`take`	take
`to_csv`	to_csv
`to_delta`	to_delta
`to_dict`	to_dict
`to_excel`	to_excel
`to_html`	to_html
`to_json`	to_json
`to_latex`	to_clipboard
`to_markdown`	to_markdown
`to_numpy`	to_numpy
`to_orc`	to_orc
`to_pandas`	to_pandas
`to_parquet`	to_parquet
`to_spark`	to_spark
`to_spark_io`	to_spark_io
`to_string`	to_string
`to_table`	to_table
`*transform`	transform	Doesn't support external reference inside lambda function
`transpose`	transpose
`truediv`	truediv
`truncate`	truncate
`unstack`	unstack
`update`	update
`values`	values
`var`	var
`where`	where
`xs`	xs